## 2 Theory
### Stochastic Gradient Boosting  
Boosting is known as an approach for enhancing the degree of precision in an estimating tool, which is carried out by repeatedly implementing the function as a series to provide a combination of outputs and weights to control the overall error of estimation. This approach has become one of the newest and most powerful learning algorithms, which are applicable in regression and classification problems [74].  
A new form of function approximation and statistical learning is [PERSON]'s SGB approach, which comes from implementing boosting in regression trees [49]. In this method, a series of relatively simple trees is determined, where each successive tree is created based on estimation residuals from the previous tree. The main source of complexity in these trees comes from two child nodes and a root node. In the SGB approach, the optimal data partitioning is calculated based on a step-by-step process; after that, the residuals of each partition are determined. Fitting a three-node tree on those residuals is the next stage for finding a new partition that will reduce the variance of the residual of the data from the aforementioned sequence of trees. According to the typical form of classification for the tree, the classification of observation data is completed, and each tree constructed is summed using this process. The combined result is used to reduce the sensitivity of this algorithm for the suspected datasets. Ensemble learning methods come from machine learning and data mining methods, which combine estimations from some algorithms through bagging, boosting, and related methods. These approaches can be formulated as follows [75]:  
\[F(x)=a_{0}+\sum_{k=1}^{K}a_{k}f_{k}(x), \tag{14}\]  
where \(K\) and \(f_{k}(x)\) are the ensemble size and base learner, respectively, for inputs \(x\) in the training dataset. In this equation, \(F(x)\) gives the ensemble estimation, which is determined via the linear combinations of estimations. Boosting approximates can be determined using an additive expansion form of the previous equation:  
\[F(x)=\sum_{k=0}^{K}\beta_{k}g(x;a_{k}), \tag{15}\]where \(g(x;a_{k})\) is chosen as a simple function in terms of \(x\) and \(a\). A forward-stage approach is employed to forecast the parameters \(a_{k}\) and the expansion coefficient of \(\beta_{k}\) for the training dataset. The first stage in this process is the initial prediction \(F_{0}(x)\), with the other \(k\) values to follow:  
\[(\beta_{k},a_{k})=argmin_{\beta,\mu}\sum_{i=1}^{N}L(y_{i},F_{k-1}(x_{i})+\beta_ {k}g(x_{i};a_{k})). \tag{16}\]  
\(L\) is [PERSON]'s function: \(L(y,F(x))=\left\{\left[y-F(x)\right]^{2}\text{ for }|y-F(x)|\leq\delta\text{ and }2\delta|y-F(x)|-\delta^{2}\text{ for }|y-F(x)|\right.\)\(>\delta\). Furthermore:  
\[F_{k}(X_{i})=F_{k-1}(x_{i})+\beta_{k}g(x_{i};a_{k}). \tag{17}\]  
Gradient boosting is used to solve Equation (16). First, a least-squares criterion is used to fit the base learning function to the pseudo-residuals:  
\[a_{k}=argmin_{a,\rho}\sum_{i=1}^{N}(y_{ik}-\rho g(x_{i};a))^{2}, \tag{18}\]  
\[y_{ik}=-\left[\frac{\partial L(y_{i},F(x_{i}))}{\partial F(x_{i})}\right]_{F( x)=F_{k-1}(x)}. \tag{19}\]  
The best coefficient is determined as follows:  
\[\beta_{k}=argmin_{\beta}\sum_{i=1}^{N}L(y_{i},F_{k-1}(x_{i})+\beta_{k}g(x_{i} ;a_{k})). \tag{20}\]  
According to this result, a hard optimization problem is changed into an easier problem by using one least-square, single-parameter optimization and the general form of a loss criterion. In the SGB algorithm, there is a focus on solving the problem using an observation that is located around the decision boundaries, which is expressed by performing the boosting operation of the model [49]. During the boosting process, it is possible to correct for the observation using an individual tree that is near to another class [76].