# Estimating the Energy Savings of Energy Efficiency Actions with Ensemble Machine Learning Models
## 3 Theoretical Modeling
### Random Forest  
RF is a supervised learning algorithm used for regression and classification tasks [33]. The basis of the RF algorithm is ensemble learning, a bagging-based learning technique that combines (averages) the individual predictions made by multiple, relatively simple ML models to produce a final prediction that is typically more accurate than the single predictions [35; 36]. The term RF stems from the two key characteristics of the algorithm, namely data and feature randomization, achieved through a technique called bootstrapping (\"random\"), and the combination of multiple decision trees (\"forest\"). Both characteristics will be presented in detail in the following paragraphs.  
The RF algorithm is one of the most indicative examples of bagging (or bootstrap aggregating), a widely used ensemble learning method that aims to train a series of models on the training data set and then to blend the predictions of these models to generate the final predictions [37]. In regression tasks, the aggregation of the individual predictions can be made with several averaging techniques, such as the mean value or the weighted average [38], while in classification tasks, the most popular method is the majority voting scheme, where the final prediction is the class that has been selected by the majority of the individual models [39].  
The structure of the RF algorithm for a regression task is presented in Figure 1. More specifically, given a data set \(D\) including \(N\) samples, each composed of \(F\) features, the algorithm trains a number of \(K\) decision trees with the following process. The RF algorithm forms \(K\) subsets of samples from the original data set \(D\) and uses them to fit \(K\) decision trees. Typically, each subset consists of \(f\leq F\) features (randomly selected from the original set of features) and \(N\) samples (randomly drawn from the original data set with replacement). Note that, according to the bootstrapping process, some of the original samples may not be selected at all in any of the constructed subsets, while others may be selected multiple times. The main reason for using bootstrapping is the reduction of variance among the predictions of the trees and the mitigation of the negative effect that outliers and particular features may have in the training process.  
RF models have many advantages that have led to their extensive use for a wide variety of tasks. First, it has been shown that RF regression models are very accurate when it comes to features with non-linear relationships and noisy data sets [40,41]. Moreover, being a tree-based method, RF models typically require little or even no data pre-processing (e.g., normalization or standardization). However, according to [42], a major drawback of these models is the lack of interpretability. Despite the fact that RF offers better predictive accuracy in comparison to single decision trees, they sacrifice the inherent interpretability of the latter that allows for verification that the developed model has captured realistic insights from the training data [43]. In addition to interpretability issues, the RF algorithm also involves more parameters than single decision trees, thus requiring more effort to be properly tuned. In this study, the Scikit-learn (Sklearn) implementation of the RF algorithm has been used [44]. The most important parameters that have been fine-tuned are the following:  
* _Number of estimators_: Number of decision trees used to provide the individual predictions.
* _Maximum depth_: Maximum possible depth for each decision tree. A deeper tree has the capacity to fit more complicated functions, but overly deep trees may result in overfitting.
* _Maximum features_: Maximum size of the random subsets of features that the decision tree takes into consideration for determining a split to a node of the tree.
* _Bootstrap_: A Boolean parameter that controls whether the model will apply bootstrapping or not.