# Estimating the Energy Savings of Energy Efficiency Actions with Ensemble Machine Learning Models
## 4 Case Study
### Results and Discussion  
The proposed ML methodology has been evaluated on a test set consisting of 754 EE investments, i.e., 20% of the whole data set. The test set includes samples from all the investment categories considered in order to evaluate the robustness of the models in different types of EE projects. The results of the experimental application are presented in Table 3. Our results indicate that all three tree-based models can effectively take into account the inherent difficulties of such a complex prediction task as the problem of estimating the energy savings of various renovation actions of different categories applied in different countries. Among the three tree-based models, XGBoost performs best in terms of _MAE_ (25,613 kWh) and _MAPE_ (32.6%), while LightGBM and RF achieve a slightly higher \(R^{2}\) score of 78.7%. On the other hand, although RF has the highest _MAE_ and _MAPE_ error scores among the three models, it is better in terms of _RMSE_ (50,459 kWh), indicating that the RF model has less significant errors compared to the other models.  
Our results indicate that models based on boosting (XGBoost and LightGBM) are generally better in terms of measures that do not penalize large errors, while RF, which based on bagging, performs better in measures like _RMSE_ that highly value large mispredictions. In this context, the proposed equal weight (EW) ensemble model outperforms all three models in all error indexes. More specifically, the EW ensemble model achieves an _RMSE_ index of 49,806 kWh, an _MAE_ index of 25,339 kWh, an _MAPE_ index of 32.2%, and an \(R^{2}\) score of 79.2%. These results confirm that the proposed ensembling model has higher predictive accuracy compared to the individual models, managing to improve the generalization error of the individual tree-based models it builds on.  
Finally, a very significant aspect of the empirical evaluation is to measure the predictive ability of the ensemble model in different investment categories. A scatter plot of the real values against the predicted values for the EW ensemble model is shown in Figure 5, where different colors are used to indicate the category of each EE project. As one may assume, some renovation categories (e.g., lighting) are expected to be easier to predict in general, while others (e.g., heating) are more demanding due to the inherent variability of the energy consumption, a fact that is visually confirmed by Figure 5 (the farther the dots are placed from the diagonal of the scatter plot, the larger the prediction error is). In this respect, Table 4 presents the _MAPE_ and \(R^{2}\) measures of the EW ensemble model per investment category. These measures have been selected for comparing different categories because they are scale-independent, contrary to _RMSE_ and _MAE_ that are not suitable for evaluating accuracy across groups of data with different scales. We find that the ensemble model performs relatively better for the cooling, lighting, and compressed air categories, producing less accurate results for the waste heat and pumps categories. This can be  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Measure** & **RF** & **XGBoost** & **LightGBM** & **EW Ensemble** \\ \hline _MAE_ (kWh) & 26,394 & 25,613 & 25,778 & **25,339** \\ _RMSE_ (kWh) & 50,459 & 50,947 & 51,002 & **49,806** \\ _MAPE_ (\%) & 33.5 & 32.6 & 32.9 & **32.2** \\ \(R^{2}\) (\%) & 78.7 & 78.3 & 78.7 & **79.2** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results on the test set. The top performing approach per measure is highlighted in boldface.  
attributed to the nature of the former types of measures, incorporating a low degree of variability regarding the achieved energy savings.