# Forecasting Seasonal Sales with Many Drivers: Shrinkage or Dimensionality Reduction?
## 4 Empirical Study
### Evaluation Design  
We evaluate the performance of our models using a rolling forecasting origin scheme. By increasing the number of forecast errors available, we increase the confidence in our findings and facilitate statistical testing. The use of a rolling origin design ensures robustness in the results. We start with a training set containing the first 139 weeks and generate 1- to 13-weeks ahead forecasts for each of the 988 SKUs. The training set is then expanded by one week and the process is repeated until week 160 giving a total of 22 forecast origins. At each forecast origin the models are re-specified automatically using the updated training data. The price and promotional plans are assumed known in the test set, as they are part of the retailer's marketing strategy. We consider different forecast horizons \(H=\{1,1-4,5-8,9-13,1-13\}\) in the comparison to take into account the different ordering and planning periods the retailer faces in practice.  
We use two error measures, the Mean Absolute Scaled Error (MASE) [66] and the Root Mean Squared Scaled Error (RMSSE) [17]:  
\[\text{MAE}_{H,s,k} =\frac{1}{H_{2}-H_{1}+1}\sum_{h=H_{1}}^{H_{2}}|y_{h,s,k}-\hat{y}_{ h,s,k}|, \tag{4}\] \[\text{MASE}_{H,s} =\frac{1}{K}\sum_{k=1}^{K}\frac{\text{MAE}_{H,s,k}}{\frac{1}{M-1 }\sum_{t=2}^{M}|y_{s,k,t}-y_{s,k,t-1}|},\] (5) \[\text{MASE}_{H} =\frac{1}{S}\sum_{s=1}^{S}\text{MASE}_{H,s}, \tag{6}\]  
\[\text{MSE}_{H,s,k} =\frac{1}{H_{2}-H_{1}+1}\sum_{h=H_{1}}^{H_{2}}(y_{h,s,k}-\hat{y}_ {h,s,k})^{2} \tag{7}\] \[\text{RMSSE}_{H,s} =\sqrt{\frac{1}{K}\sum_{k=1}^{K}\frac{\text{MSE}_{H,s,k}}{\frac{ 1}{M-1}\sum_{t=2}^{M}(y_{s,k,t}-y_{s,k,t-1})^{2}}},\] (8) \[\text{RMSSE}_{H} =\frac{1}{S}\sum_{s=1}^{S}\text{RMSSE}_{H,s}, \tag{9}\]  
Figure 2: Distribution of the average weekly sales on promotional and non-promotional weeks by category.  
where \(y_{h,s,k}\) is the \(h\)th observed value in the forecast horizon \(H\) of the SKU \(s=1,\ldots,S\) at the \(k=1,\ldots,K\) forecast origin, and \(\hat{y}_{h,s,k}\) is the corresponding forecast. Once the Mean Absolute Error (MAE) and Mean Squared Error (MSE) are calculated, they are summarised across forecast origins and then across SKUs. Both MASE and RMSSE are scale-independent and hence suitable for comparing the forecasts across multiple products of different scales and units. This is achieved by scaling the forecasts errors by the MAE or MSE of the 1-step ahead in-sample naive forecast errors, to match the absolute or quadratic loss of the numerator [67]. Squared errors favour forecasts that track the mean of the target series, which is influenced by the various special events and promotions, while absolute errors favour forecasts that track the median of the target series and hence focus on the structure of the data.  
We can also use the MAE and MSE errors to perform tests on the statistical significance of any reported differences. To this end we use the Multiple Comparison with the Best (MCB) test, advocated by [68]. This is a restricted version of the non-parametric Nemenyi test [69], focusing only on testing the best performing model against the rest. The test is implemented using the nemenyi() function of the R package tsutils[70]. Finally, as part of the testing, we also report mean ranks of the various forecasts.