# Estimating the Energy Savings of Energy Efficiency Actions with Ensemble Machine Learning Models
## 3 Theoretical Modeling
### Light Gradient Boosting Machine  
LightGBM [51] is another GB framework, similar to XGBoost in the sense that it is based on the GB algorithm and uses decision trees as weak learners. LightGBM was originally implemented by Microsoft, focusing on performance, scalability, and optimal memory usage, and it has been widely exploited for ranking, classification, and regression ML tasks [52].  
Although LightGBM is similar to XGBoost in terms of supporting parallel processing capabilities, similar loss functions, and regularization parameters, it differs significantly in the way that decision trees are constructed. Most GB algorithms perform a depth-wise growth of the decision trees that are developed. On the contrary, LightGBM has adopted a leaf-wise method for growing the trees, selecting the leaf that has the best possible loss decrease regardless of its level of depth [53]. This method usually results in lower loss managing to better fit to the training data. Another structural difference between the two frameworks is that LightGBM uses a different method for determining the optimal split point of the decision trees [54]. Most GB algorithms, including XGBoost, search the optimal split point on the sorted feature values. However, LightGBM uses a histogram-based learning algorithm, which is more efficient and demands less memory.  
The efficiency of LightGBM is mainly due to the use of two state-of-the-art techniques, namely gradient-based one-side sampling and exclusive feature bundling, which are combined together to form an powerful model training framework with increased accuracy compared to most GB algorithms. On the one hand, gradient-based one-side sampling is a sampling technique that is used to select samples based on their gradients. This technique aims to exploit the fact that instances with smaller gradient are better trained compared to instances with larger gradients that are considered under-trained. Thus, sampling instances with larger gradients are highly beneficial to quicker information gain. On the other hand, according to [51], exclusive feature bundling is a novel technique that aims to reduce the number of features by regrouping mutually exclusive features into bundles, treating them as a single feature. This technique has been proven particularly effective when the feature space is sparse and several features are almost exclusive, i.e., they are usually zero at the same time. A clear example of such feature spaces are tasks that involve many categorical variables that are transformed through one-hot encoding.  
The most important features that have been considered for fine-tuning the LightGBM model, implemented using the official LightGBM library for Python [51], are the following:* _Number of estimators_: Number of boosted decision trees fit during the training process.
* _Maximum depth_: Similar to RF and XGBoost, this hyper-parameter is used to limit the depth of the grown decision tree. Because of LightGBM's lead-wise structure, fine-tuning this parameter is critical to avoid overfitting.
* _Number of leaves_: This hyper-parameter affects the complexity of the tree model. When the number of leaves is set to \(2^{Maximum\ depth}\), then the tree has the same number of leaves as a depth-wise tree, which is highly discouraged. On the other hand, if the number of leaves is not restricted, the model is prone to overfitting.
* _Learning Rate_: Similar to XGBoost, this hyper-parameter affects the learning rate of the boosting process.