# Automotive OEM Demand Forecasting: A Comparative Study of Forecasting Algorithms and Strategies
## 4 Experiments and Results  
In this section, we describe the experiments we conducted (summarized in Table 4) and assess their results with metrics and criteria we described in Section 3.5. We summarize the outcomes in Table 5, to understand if a particular model performs significantly better than others. To evaluate the models, we used nested cross-validation [71], which is frequently used to evaluate time series models. To ensure conditions on ML streaming models were comparable to ML batch models, we implemented the nested cross-validation evaluation strategy. By doing so, we ensured the streaming model did not see new events until the required month was predicted. In order to test the models, we set apart the last six months of data. We published the nested cross-validation implementation for streaming models it in the following repository: [[https://github.com/JozefStefanInstitute/scikit-multiflow](https://github.com/JozefStefanInstitute/scikit-multiflow)]([https://github.com/JozefStefanInstitute/scikit-multiflow](https://github.com/JozefStefanInstitute/scikit-multiflow)) (accessed date 21 July 2021). In Experiments 1-4, we assessed how events in year 4 affected model learning and if they significantly degraded forecasts. We also compared two different sets of features, resulting from two different procedures to obtain them. We obtained the best performance with local models trained over the last three years of data. Removing features with high collinearity did not enhance the median of R\({}^{2}\)adj and MASE. Therefore, we consider Experiment 3 performed best, having the best MASE and R\({}^{2}\)adj values. In contrast, the rest of the evaluation criteria values were acceptable. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Years of Data** & **Experiment** & **Feature Selection** & **Number of Features** \\ \hline \multirow{2}{*}{All years available} & Experiment 1 & I & 6 \\  & Experiment 2 & II & 6 \\ \hline \multirow{6}{*}{Last three years} & Experiment 3 & I & 6 \\  & Experiment 4 & II & 6 \\  & Experiment 5 & II & 6 \\  & Experiment 6 & II & 6 \\  & Experiment 7 & II & \(\sqrt{N}\) \\  & Experiment 8 & II & \(\sqrt{N}\) \\  & Experiment 9 & II & \(\sqrt{N}\) \\  & Experiment 10 & II & \(\sqrt{N}\) \\  & Experiment 11 & Only past demand & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Description of experiments performed. Regarding the feature selection procedure, we consider two cases: (I) top features ranked by a GBRT model and curated by a researcher, and (II) top features ranked by a GBRT model, removing those with strong collinearity, curated by a researcher as well. N in the _“Number of features”_ column refers to the number of instances in a given dataset. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Experiment** & **R\({}^{2}\)adj** & **MASE** & **5\% Error** & **10\% Error** & **20\% error** & **30\% Error** & **UE** & **90\%+ Error** \\ \hline \multirow{2}{*}{**Experiment 1**} & 0.8584 & 1.1450 & **0.0670** & 0.1086 & 0.2039 & 0.3051 & **0.3854** & 0.4077 \\  & 0.8447 & 1.1450 & 0.0655 & 0.1101 & 0.1920 & 0.2887 & 0.4182 & 0.3928 \\  & **0.9067** & 0.9150 & 0.0655 & **0.1280** & **0.2351** & 0.3095 & 0.4256 & 0.3928 \\  & 0.8998 & 0.9750 & 0.0655 & 0.1176 & 0.2143 & 0.3051 & 0.4152 & 0.4018 \\  & 0.8757 & 0.3900 & 0.0536 & 0.1116 & 0.2173 & 0.3140 & 0.4762 & 0.3497 \\  & 0.8679 & 0.3350 & 0.0565 & 0.1012 & 0.1875 & 0.2768 & 0.4851 & 0.3601 \\  & 0.8903 & 0.3550 & 0.0521 & 0.1131 & 0.2247 & **0.3155** & 0.4851 & **0.3408** \\  & 0.8786 & **0.3100** & 0.0506 & 0.0938 & 0.1890 & 0.2813 & 0.4658 & 0.3497 \\  & 0.70161 & 0.8100 & 0.0357 & 0.0714 & 0.1428 & 0.2143 & 0.7321 & 0.3601 \\  & \(-\)1.5344 & 0.5300 & 0.0178 & 0.0536 & 0.1250 & 0.2143 & 0.7143 & 0.4613 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Median of results obtained for each ML experiment. We abbreviate _under-estimates_ as _UE_. In Experiments 9–10, streaming models based on Hoeffding bound show poor performance, resulting in negative R\({}^{2}\)adj values.