# GlassBoost: A Lightweight and Explainable Classification Framework for Tabular Datasets
## 3 Decision Trees and Gradient-Boosting Machines (GBMs)
### Decision Tree Ensembles  
Decision tree ensemble models leverage the power of multiple decision trees to enhance prediction performance. While individual trees may exhibit limitations, combining their predictions through a suitable aggregation method, such as boosting, can significantly improve accuracy and robustness. An ensemble of weak learners can be written as Equation (1). \[\hat{y}_{i}=\sum_{k=1}^{K}f_{k}(x_{i}),f_{k}\in\mathcal{F}. \tag{1}\]  
Here, \(K\) is the number of trees in the ensemble, and \(f_{K}\) is a hypothesis (function) within the hypothesis (function) space \(\mathcal{F}\). In this context, \(\mathcal{F}\) denotes the set of all possible CART models, and \(\hat{y}_{i}\) represents the predicted value of the actual label of \(x_{i}\) across trees in the ensemble. The objective function of an ensemble of learners in general is written as Equation (2). \[\mathcal{L}=\sum_{i}^{n}l(y_{i},\hat{y}_{i})+\sum_{k=1}^{K}\omega(f_{k}). \tag{2}\]  
where \(\omega(f_{k})\) is the regularisation term that measures the complexity of the tree \(f_{K}\), defined later in this section, and \(\;l(y_{i},\hat{y}_{i})\) is the loss function that quantifies the error or discrepancy between \(y\) and \(\hat{y}\). Although Equation (2) provides the general form of the objective function for ensembles, it is not directly applicable to gradient-boosting machines (GBMs). In gradient boosting, each tree in the ensemble is trained sequentially to correct the errors of its predecessors. This requires an additive training approach because optimisingthe entire ensemble simultaneously is far more complex than standard optimisation problems, where gradients can be easily computed. Jointly learning all trees is mathematically intractable, as no feasible classical method exists to compute the gradients for the entire ensemble at once. Therefore, in GBMs, at time step \(t\), the predictions from previous trees are considered fixed, and a new tree is added incrementally. The prediction value at step \(t\) is denoted as \(\hat{y}_{i}^{(t)}\). The objective function for a GBM at time step \(t\) can be written as Equation (3). \[\mathcal{L}^{(t)}=\sum_{i=1}^{n}l(y_{i},\hat{y}_{i}^{(t)})+\omega(f_{t}). \tag{3}\]  
The first term of Equation (3) is the loss function that measures the error between \(y_{i}\) and \(\hat{y}_{i}^{(t)}\), and \(n\) is the number of training data points. The second term of the loss function, \(\omega(f_{t})\), is the regularisation term that imposes a penalty on the complexity of the tree \(f_{t}\). As shown in Equations (4) and (5), the prediction at time \(t\) is computed recursively, relying on predictions from previous time steps. \[\hat{y}_{i}^{(0)}=0, \tag{4}\]  
\[\hat{y}_{i}^{(t)}=\sum_{k=1}^{t}f_{k}(x_{i})=\hat{y}_{i}^{(t-1)}+f_{t}(x_{i}). \tag{5}\]  
By substituting Equation (5) into Equation (3), the objective function can be expressed as follows:  
\[\sum_{i=1}^{n}l(y_{i},\hat{y}_{i}^{(t-1)}+f_{t}(x_{i}))+\omega(f_{t}). \tag{6}\]  
Expanding the loss function, \(l(y_{i},\hat{y}_{i}^{(t)})\), around \(l(y_{i},\hat{y}_{i}^{(t-1)})\) using a second-order Taylor expansion yields the following:  
\[ \begin{split} l(y_{i},\hat{y}_{i}^{(t)})&\simeq l(y _{i},\hat{y}_{i}^{(t-1)})+\\ &\frac{\partial l}{\partial\hat{y}_{i}^{(t-1)}}\Bigg{|}_{\hat{y}_ {i}^{(t-1)}}(\hat{y}_{i}^{(t)}-\hat{y}_{i}^{(t-1)})+\frac{1}{2}\frac{\partial ^{2}l}{\partial(\hat{y}_{i}^{(t-1)})^{2}}\Bigg{|}_{\hat{y}_{i}^{(t-1)}}(\hat{y }_{i}^{(t)}-\hat{y}_{i}^{(t-1)})^{2}.\end{split}  \tag{7}\]  
Applying Equation (5) to Equation (7), we can rewrite Equation (7) as follows:  
\[ \begin{split} l(y_{i},\hat{y}_{i}^{(t)})&\simeq l(y _{i},\hat{y}_{i}^{(t-1)})+\frac{\partial l}{\partial\hat{y}_{i}^{(t-1)}}f_{t}( x_{i})\\ &+\frac{1}{2}\frac{\partial^{2}l}{\partial(\hat{y}_{i}^{(t-1)})^{ 2}}f_{t}^{2}(x_{i}).\end{split}  \tag{8}\]  
Equation (8) can then be abstracted as Equation (9) by defining \(g_{i}\) and \(h_{i}\), as demonstrated in Equations (10) and (11), respectively.