# Robust Sales forecasting Using Deep Learning with Static and Dynamic Covariates
## 3 Empirical Study
### Evaluation Design  
DeepAR global models were trained using the M5 dataset, consisting of 30,490 sales of products across the 10 stores.  
The information about the short-term sales trend was derived using the sales data from the previous 28 days as inputs. We adopted the M5 competition's framework, where the last 28 days of each time series (from 23 May 2016 to 19 June 2016) were reserved as a testing set for out-of-sample evaluation. The remaining data, spanning from 29 January 2011 to 22 May 2016 (a total of 1941 days), was used for training the models. In order to reach good accuracy results, it is essential to identify a high-performing model during the testing phase. Usually, a validation set is utilized to assess the most appropriate model. The success of a deep learning model is significantly impacted by various factors, including the values for hyperparameters and the values for the initial weights. In order to identify the best model, we used the final 28 days of in-sample training data, covering the period from 25 April 2016 to 22 May 2016, as the validation set. The hyperparameter optimization process was carried out using the Optuna optimization framework [25], with the root mean squared error (RMSE) [7] serving as the accuracy metric for model selection. To evaluate the significance of the different types of features, we included them individually and in various combinations, and compared their performance to the baseline case where no features were used.  
We assessed the performance of the DeepAR models using two metrics commonly employed in the forecasting literature [26]: the root mean squared scaled error (RMSSE) and the mean absolute scaled error (MASE):  
\[\text{RMSSE}_{i}=\sqrt{\frac{\frac{\frac{1}{h}\sum_{t=n+1}^{n+h} \left(\text{z}_{i,t}-\text{z}_{i,t}\right)^{2}}{\frac{1}{n-1 }\sum_{t=2}^{n}\left(\text{z}_{i,t}-\text{z}_{i,t-1}\right)^{2}}}}, \tag{5}\]  
\[\text{MASE}_{i}=\frac{\frac{\frac{1}{h}\sum_{t=n+1}^{n+h}\left| \text{z}_{i,t}-\text{z}_{i,t}\right|}}{\frac{1}{n-1 }\sum_{t=2}^{n}\left|\text{z}_{i,t}-\text{z}_{i,t-1}\right|} \tag{6}\]  
where z\({}_{i,t}\) is the sales of item \(i\) at time \(t\) and z\({}_{i,t}\) is its forecast, \(n\) is the length of the training set and \(h\) is the forecast horizon. In this particular case study, the time frame considered for the forecast horizon is 28 days. Once calculated for each item, the RMSSE and the MASE were summarized across all items.  
The RMSSE was the metric used to assess the accuracy of point forecasts in the M5 competition [27]. RMSSE and MASE are two scale-independent measures that can be utilized to compare forecasts across different items with different scales and units. This is because they scale the forecast errors based on the MSE and MAE of the 1-step ahead in-sample naive forecast errors, respectively. This ensures that the errors are measured in terms of their absolute or squared magnitude, which allows for fair comparisons across different products. Additionally, RMSSE and MASE differ in how they weight the errors. RMSSE uses squared errors, which emphasizes forecasts that closely adhere to the mean of the target series. MASE uses absolute errors, which emphasizes forecasts that closely adhere to the median of the target series. This difference in weighting allows for different perspectives on the underlying structure of the data.