# Conditional Temporal Aggregation for Time Series Forecasting Using Feature-Based Meta-Learning
## 4 Empirical Evaluation
### Experimental Design  
Our experimental design closely follows the original setup of each competition. In this regard, in the M4 data set we consider a forecasting horizon of 1.5 years, using the last 18 months as a test set and the rest of the observations for validation and training, i.e., for defining the values of the hyperparameters to be used by the meta-leaner and estimating the model accordingly. Similarly, in the M5 data set, we use the last 28 days as a test set and the previous observations for training and validation.  
In the M4 data set, forecasting accuracy is measured using the mean absolute scaled error (MASE; [57]), defined as follows:  
\[\text{MASE}=\frac{1}{h}\frac{\sum\limits_{t=n+1}^{n+h}|y_{t}-\hat{y}_{t}|}{ \frac{1}{h-m}\sum\limits_{t=m+1}^{n}|y_{t}-y_{t-m}|},\]  
where \(y_{t}\) is the actual observation of the series at period \(t\), \(\hat{y}_{t}\) is the respective forecast, \(h\) is the forecasting horizon, \(n\) is the number of the in-sample observations (data points available in the train set), and \(m\) is the number of periods within a seasonal cycle (e.g., \(m=12\) for monthly data). The values of MASE are averaged across different time series to estimate overall accuracy. Lower MASE values suggest better forecasts.  
MASE is independent of the scale of the data, has a predictable behavior, has a defined mean and a finite variance, and is symmetric in the sense that it penalizes equally positive and negative forecast errors, as well as large and small ones. Therefore, it is preferred over other popular accuracy measures, such as relative or percentage errors. Moreover, MASE was one of the two measures used in M4 for evaluating the methods originally submitted in the competition.  
In a similar fashion, in the M5 data set, we measure accuracy using the root mean squared scaled error (RMSSE), defined as follows:  
\[\text{RMSSE}=\sqrt{\frac{1}{h}\frac{\sum\limits_{t=n+1}^{n+h}(y_{t}-\hat{y}_{t })^{2}}{\frac{1}{n-m}\sum\limits_{t=m+1}^{n}(y_{t}-y_{t-m})^{2}}}.\]  
RMSSE shares the same properties as MASE, but is more suitable for data that display intermittency. This is because squared errors are optimized for the mean of the data [58] instead of the median [59]. Therefore, methods that report lower RMSSE values will identify more accurately the expected demand, avoiding putting too much focus on zero sales. Additionally, this measure is consistent to the one originally used in M5 for evaluating the accuracy of the submitted methods.  
The validation of the proposed meta-learner took place using the 18 and 28 observations preceding the test sets of the M4 and M5 data sets, respectively. As explained in Section 3.3, various sets of hyperparameter values were considered, and the top-performing one on the validation set was selected per case to train the meta-learner. The selected hyperparameter values are presented in Table 2.  
Note that the monthly series of the fast-moving data set can be naturally aggregated at six different levels to form a meaningful temporal hierarchy, considering buckets of 1 (month), 2 (bi-month), 3 (quarter), 4 (four months), 6 (semi-year), and 12 (year) observations. The same is not true for the daily series of the slow-moving data set since different months may consist of a different number of days. In this respect, we considered four temporal levels using buckets of 1 (day), 7 (week), 14 (two weeks or about half a month), and 28 observations (four weeks or about a month) that offer a reasonable temporal hierarchy. As a result, the classifier of the meta-learner will be trained in the M4 and M5 data sets by considering six and four aggregation levels as labels, respectively.