## 4 Empirical Study
### Results
Beginning with 48 observations in the training set (data from January 2007 to December 2010) and finishing with 63, a new ARIMA model using the [PERSON] (2010) algorithm was identified in each step and used to forecast the next observation that is not in the training data. Then, error measures calculated from the 16 one-step forecasts were obtained. The same procedure was applied to the [PERSON] and [PERSON] (1996) statistical framework, i.e., a new ETS model was identified in each step of the time series cross-validation. In the case of the seasonal naive method, in each step, the forecast of the next observation that was not in the training data was equal to the last observed value from the same month of the previous year. The Ljung-Box test using all of the available data was also performed for these three forecasting approaches and the \(p\)-values obtained are shown in Table 1. Since there is no universally agreed-upon performance measure that can be applied to every forecasting situation, multiple criteria are therefore often needed to give a comprehensive assessment of forecasting models ([PERSON] and [PERSON], 2010). The RMSE, the MAE, and the MAPE are the most commonly used forecast error measures among both academics and practitioners ([PERSON], 2007). We also consider the MASE in order to overcome some of the limitations of the MAPE. According to the results for this case study, our procedure to find an ARIMA model produces the most accurate forecasts when judged by the four most common performance measures RMSE, MAE, MAPE and MASE (with the single exception of MAPE for the Sandals time series). The RMSE, MAE and MASE produce the same accuracy ranking, with the exception for the Booties time series--when considering MAE, our procedure to find a state space model is more accurate than the seasonal naive method, but when considering RMSE, the opposite happens, although according to MASE, both approaches have the same performance. The MAPE consistently ranks differently, which reinforces the impact that its limitations can have on the results. Our procedure to find a state space model and an ARIMA model consistently forecasts more accurately than the [PERSON] and [PERSON] (1996) statistical framework and the [PERSON] and [PERSON] (2010) algorithm. The improvements in forecast accuracy over the [PERSON] and [PERSON] (2010) algorithm are quite considerable: for Boots time series, the RMSE and the MAE (MASE) are 65% and 70% smaller, respectively; for Bootes time series, the RMSE and the MAE (MASE) are 45% and 42% smaller, respectively; for Flats time series, the RMSE and the MAE (MASE) are 29% and 22% smaller, respectively; for Sandals time series, the RMSE and the MAE (MASE) are 69% and 67% smaller, respectively; for Shoes time series, the RMSE and the MAE (MASE) are 36% and 32% smaller, respectively. The improvements in forecast accuracy over the [PERSON] and [PERSON] (1996) statistical framework are also significant: for Boots time series, the RMSE and the MAE (MASE) are 23% and 20% smaller, respectively; for Bootes time series, the RMSE and the MAE (MASE) are 30% and 13% smaller, respectively; for Flats time series, the RMSE and the MAE (MASE) are 11% and 7% smaller, respectively; for Sandals time series, the RMSE and the MAE (MASE) are 48% and 33% smaller, respectively; for Shoes time series, the RMSE and the MAE (MASE)are 34% and 35% smaller, respectively. Our procedure to find an ARIMA model also outperforms significantly the seasonal naive method: for Boots time series, the RMSE and the MAE (MASE) are 55% and 48% smaller, respectively; for Boots time series, the RMSE and the MAE (MASE) are 1% and 11% smaller, respectively; for Flats time series, the RMSE and the MAE (MASE) are 49% and 51% smaller, respectively; for Sandals time series, the RMSE and the MAE (MASE) are 21% and 5% smaller, respectively; for Shoes time series, the RMSE and the MAE (MASE) are 52% and 60% smaller, respectively.