# Estimating the Energy Savings of Energy Efficiency Actions with Ensemble Machine Learning Models
## 3 Theoretical Modeling
### Extreme Gradient Boosting  
Gradient boosting (GB) is another type of ensemble learning algorithm used for both regression and classification tasks [45]. The aim of GB, similar to other boosting techniques, is to construct a \"strong learner\" model by iteratively combining several \"weak learner\" models, each constructed so that the prediction error of the previous model is decreased (in each stage of the algorithm, a decision tree--regressor or classifier according to the task--is fit on the negative gradient of the loss function.) [46]. Since the new models built are specialized in tackling the particular issues of the previous ones, the overall performance  
Figure 1: The structure of the random forest algorithm, which is based on bootstrap aggregating (or bagging) the predictions of multiple decision tree models. of the final model is effectively improved. Once all individual trees have been added to the ensemble, the final model is used to predict new samples. The overall architecture of the GB algorithm is presented in Figure 2. The GB ensemble model is composed of \(K\) decision trees. The first decision tree is trained using the input feature matrix \(X\) and the original label \(y\). After the predictions of the tree have been generated, the residual errors \(E1\) are calculated. The second decision tree is then trained using the \(E1\) errors as labels instead of the original \(y\) ones. The next trees are trained following the same process in an iterative fashion. Finally, when the last individual tree is trained, the training process terminates, and the complete set of trees is used for making future predictions. In this context, XGBoost belongs to the family of GB methods but differs from them in the sense that it uses advanced regularization, namely \(L1\) and \(L2\) values, to enhance the generalization capabilities of the developed models. XGBoost is based on an efficient implementation of the GB algorithm, delivering higher computational performance in comparison to traditional GB models and offering reduced training times through parallelization across clusters [47]. Note that XGBoost, like RF, also constructs ensembles from decision tree models; the main difference is that in XGBoost, the trees are added sequentially to the ensemble model and fitted in an attempt to minimize the prediction errors of the previous models (this is also the main difference between bagging and boosting). Finally, the name \"gradient boosting\" is due to the gradient descent optimization algorithm used to minimize the error. XGBoost has several advantages that have established it as one of the most sovereign models for GB. As mentioned in the previous paragraph, it has a customized way of pruning the decision trees that form the ensemble, resulting in faster training and better ability to process high volume data sets [48]. Among its optimizations, XGBoost uses an approximate greedy algorithm of weighted quantiles during the process of splitting the nodes of the trees instead of comparing all different possible splits. XGBoost also divides data to smaller segments to enable parallel processing of samples [49], while it exploits the cache memory to save gradients and enable faster calculations [50]. The above-mentioned features have led to its wide adoption during the last decade for classification and regression tasks in many applications. Figure 2: The general structure of the gradient boosting algorithm, which is based on boosting the predictions of multiple decision tree models. Similar to the case of RF, XGBoost is also based on a set of hyper-parameters that are crucial for achieving high-accuracy predictions. These hyper-parameters are related not only to the GB process, including elements such as the number of trees that should be used during training, but also to the structure of each individual tree and adjusting parameters, such as the depth of the tree and the minimum loss reduction required to make a further partition on a leaf node of the tree.