# Hyperparameter Optimization of Ensemble Models for Spam Email Detection
## 3 Methodology
### Methods and Machine Learning Classifiers
#### 3.3.2 XGBoost  
Extreme gradient boost (XGBoost) is a supervised ensemble machine-learning algorithm. The ensemble technique used by the algorithm is the boosting technique. The algorithm trains Classification and Regression Tree (CART) learners sequentially. Models are trained one after another, and the succeeding models are targeted at optimizing the loss function or residuals of the previous models in a bid to improve predictability. The algorithm works by calculating the similarity scores (Equation (1)) of the residuals of the independent email features at each node of the base CART [35].  
\[SS=\frac{\sum_{i=1}^{n}\left(y_{i}-f(x_{i})\right)^{2}}{n*\lambda} \tag{1}\]  
\(SS=Similarity\ Score\)  
\(y_{i}=the\ value\ of\ the\ ith\ variable\ to\ be\ predicted\)  
\(f(x_{i})=the\ current\ prediction\ for\ independent\ features\)  
\(n=the\ number\ of\ residuals\)  
\(\lambda=regularization\ parameter\)  
The similarity score of each node within the tree is used to determine the information gain (Equation (2)).  
\[G=SS_{a}-SS_{b} \tag{2}\]  
\(SS_{a}=Similarity\ Score\ after\ split\)  
\(SS_{b}=Similarity\ Score\ before\ split\)  
The regularization parameter \(\lambda\), reduces the effect of outliers and controls the pruning of the trees within the model. A higher value of \(\lambda\) will reduce the value of the similarity score which results in a smaller information gain (Equation (2)). The gain is compared with gamma \(\gamma\), another important parameter used in the XGBoost algorithm. Gamma determines if there will be a split at a node. If gamma (\(\gamma\)) is less than gain (\(G\)), there will be a split at the node else, there will be no split. This serves as a mechanism to control overfitting. A high gamma would result in an extremely pruned tree. In this study, R default values of 0 and 1 were retained for \(\gamma\) and \(\lambda\), respectively.  
The prediction of the base CART model based on the selected leaf node is determined as:  
\[Output=\frac{\sum_{i=1}^{n}(y_{i}-f(x_{i}))}{n*\lambda} \tag{3}\]  
The sum of residuals is divided by the product of the number of residuals and the regularization parameter, \(\lambda\). The predictions of successive CART models are calculated as in Equation (4).  
\[New_{p}=Previous_{p}+(eta*Output) \tag{4}\]  
\(New_{p}=the\ prediction\ of\ the\ current\ CART\ learner\)  
\(Previous_{p}=the\ prediction\ of\ the\ Previous\ CART\ learner\)  
\(eta=this\ is\ the\ learning\ rate\)  
The output (Equation (3)) is the prediction of the base CART learner. The learning rate or eta is another important parameter that reduces the feature weight to prevent overfitting. It also determines how quickly a model will converge. The default value for this parameter in R is 0.3.  
The workflow of the XGBoost algorithms discussed in Equations (1)-(4) can be summarized in seven steps as thus:  
Step 1: Initialization--A set of decision-tree models is trained to classify a small set of emails as spam or not spam.  
Step 2: Boosting--New models are added to the ensemble and trained to correct mistakes made by previous models by focusing on misclassified emails.  
Step 3: Gradient Descent--Parameters of new models are optimized by minimizing the ensemble's loss function using gradient descent.  
Step 4: Regularization--Regularization is used to prevent overfitting by penalizing models for having too many parameters or complex decision boundaries.  
Step 5: Pruning--The decision tree is pruned by removing leaves with low weight to prevent overfitting and improve generalization.  
Step 6: Repeat--Steps 2 through 5 are repeated for a fixed number of iterations or until a stopping criterion is met, such as reaching a certain level of accuracy or the number of boosting rounds.  
Step 7: Return--The final ensemble of base models is returned as the final model and can be used to classify new emails as spam or not spam by taking a majority vote of the base models.