# Estimating the Energy Savings of Energy Efficiency Actions with Ensemble Machine Learning Models
## 4 Case Study
### Accuracy Measures  
The selection of the most appropriate accuracy measures is a necessary step for the evaluation of any regression model. In this study, we evaluate the accuracy of the four models (the three models presented in Section 3 and the ensemble model) based on the mean absolute error (\(MAE\)), the mean absolute percentage error (\(MAPE\)), the root mean squared error (\(RMSE\)), and the coefficient of determination (\(R^{2}\)) between the forecasts and the real values [59].  
\(MAE\) is the average of all absolute errors between the real and the predicted value, thus being relatively robust to outliers and easy to interpret, as it is expressed at the same scale as the actual observations. On the other hand, \(RMSE\) gives a relatively higher weight to larger errors and therefore helps in the detection of large mispredictions by the models. Both measures are very useful for the interpretation of the results, but when it comes to comparing different investment categories (which have different scales of energy savings), the use of scale-independent measures is necessary. \(MAPE\) is such a measure that incorporates scale-independence and high interpretability. Moreover, since the data set does not include any zero values that affect the ability to calculate \(MAPE\), this measure is highly beneficial for comparing the accuracy of the models on different investment categories. Finally, \(R^{2}\) is a statistical measure that shows the proportion of variance the predictions explain over the target variable, being an indicator of how well the data fit the regression models. The formulas for computing the above-mentioned measures are presented as follows:  
\[MAE=\frac{1}{n}\sum_{1}^{n}|y-\hat{y}|, \tag{1}\]  
\[RMSE=\sqrt{\frac{1}{n}\sum_{1}^{n}(y-\hat{y})^{2}}, \tag{2}\]  
\[MAPE=\frac{100\%}{n}\sum_{1}^{n}\Big{|}\frac{y-\hat{y}}{y}\Big{|}, \tag{3}\]  
\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Model** & **Hyper-Parameter** & **Min Value** & **Max Value** & **Step** & **Selected** \\ \hline RF & Number of estimators & 100 & 500 & 100 & 100 \\  & Maximum depth & 1 & 7 & 2 & 3 \\  & Maximum features & 0.2 & 1 & 0.2 & 1 \\  & Bootstrap & & Values tested: True, False & True \\ \hline XGBoost & Number of estimators & 100 & 500 & 100 & 300 \\  & Maximum depth & 2 & 10 & 2 & 6 \\  & Learning rate & 0.1 & 0.9 & 0.2 & 0.3 \\  & Complexity control & 0 & 10 & 2 & 0 \\ \hline LightGBM & Number of estimators & 20 & 100 & 20 & 80 \\  & Maximum depth & 2 & 10 & 2 & 6 \\  & Number of leaves & 10 & 50 & 10 & 30 \\  & Learning rate & 0.02 & 0.1 & 0.02 & 0.08 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The search space and the selected values of the hyper-parameters for the tree-based regression models considered for predicting energy savings.  
\[R^{2}=1-\frac{\frac{\sum_{1}^{n}(y-\hat{y})^{2}}{\sum_{1}^{n}(y-\hat{y})^{2}}}, \tag{4}\]  
where \(y\) and \(\hat{y}\) are the real and the predicted value of the energy savings for the examined EE investment in the test set, respectively, \(\hat{y}\) is the average value of the real observations, and \(n\) is the total number of the samples included in the evaluation.