## 3 Machine Learning Models
### Decision Trees and Random Forests  
Another major category of machine learning algorithms applied to the problem of time series forecasting is the set of tree-based models. This category, although underrepresented in the scientific literature in comparison to its deep learning counterparts, is at the forefront of many recent forecasting competitions' leaderboards [27].  
In time series forecasting, decision trees form a succession of binary decisions associated to supervised features of the used dataset, which drive the forecasting procedure based on the minimization of the data variance [28]. The modeling of the decision tree for time series is based on the classification and regression tree algorithm (CART), developed by [PERSON] et al. [29].  
Random forests are formed by aggregating the decisions of a number of trees on a specific problem. In time series forecasting, the use of random forest regression algorithms has the advantage of a lower model variance in relation to other machine learning approaches [30]. Two of the most prominent models that also display superior performance in forecasting competitions are XGBoost and Lightgbm. XGBoost constitutes an implementation of the Gradient Boosting learning method, namely that it combines a number of weak learners into a strong learner in a linear fashion. Both CART and linear classifiers can be used as base models for the XGBoost algorithm, while a second-order Taylor expansion of the cost function entails an enhanced information profile. The Lightgbm model is another implementation of the Gradient Boosting Decision Tree algorithm (GBDT), based on two novel techniques: gradient-based one-side sampling and the exclusive feature bundling [31].  
In the scientific literature presented in later sections of this work, the XGBoost algorithm appears in comparison to ARIMA and deep learning models in the work by [PERSON] et al. [32]. In this paragraph, we will present some comparative results derived from the popular Makridakis forecasting competitions, as well as from the recently published scientific literature.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Article** & **Algorithms** & **Metrics** & **Dataset** \\ \hline [PERSON] et al. [33] & ARIMA\((0,1,1)\times(0,1,1)_{12}\) & MAE & infectious disease prediction \\  & XGBoost & RMSE & \\  & MAPE & & \\ \hline [PERSON] et al. [34] & ARIMA\((3,1,0)\times(1,1,0)_{12}\) & MAE & infectious disease prediction \\  & XGBoost & MPE & \\  & MAPE & & \\  & RMSE & & \\  & & MASE & \\ \hline [PERSON] et al. [35] & ARIMA\((0,1,1)\times(0,1,1)_{7}\) & MAE & COVID-19 confirmed cases \\  & XGBoost & RMSE & \\  & MAPE & & \\ \hline [PERSON] et al. [36] & ARIMA\((0,1,1)\) & MAE & annual rice production \\  & XGBoost & MPE & \\  & RMSE & & \\  & MAPE & & \\ \hline  & ARIMA & RMSE & retail sales volume \\  & LSTM & MAE & \\  & Prophet & & \\  & GBDT & & \\  & XGBoost & & \\ \hline [PERSON] et al. [38] & ARIMA\((2,1,1)\) & MSE & short time load forecasting \\  & SARIMA & RMSE & \\  & LSTM & MAE & \\  & LightGBM & & \\  & Prophet & & \\  & VAR & & \\ \hline [PERSON] et al. [39] & various & various & retail unit sales (M5) \\  & & & multiple categories (M4) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Summary of ARIMA and Decision Tree-based models’ Comparison Studies in Time Series Forecasting.  
Although limited, the scientific literature focused on the comparison of ARIMA approaches with decision tree-based forecasting algorithms expands on a variety of applications. It is, however, interesting to note that the majority of forecasting applications using tree-based approaches consist of hybrid models, which we will refer to in the section on hybrid ARIMA models: