# Machine Learning Ship Classifiers for Signals from Passive Sonars
## 3 Classifier Models
### eXtreme Gradient Boosting  
XGBoost builds an incremental model and progressively enables optimization of arbitrary differentiable loss functions. At each stage, \(n\) regression trees are adjusted in the direction opposite to the gradient of the loss function. Binary classification is a special case in which only a single regression tree is used [13]. XGBoost is an ensemble learning method in which multiple weak learners, such as decision trees, are trained sequentially, and the model is optimized from the errors made by previous learners. The objective function in XGBoost comprises two terms:  
\[\text{Obj}(\theta)=\sum_{i=1}^{n}L(y_{i},\hat{y}_{i})+\sum_{k=1}^{K}\Omega(f_ {k}),\]  
where \(L(y_{i},\hat{y}_{i})\) is the loss function that measures the extent to which prediction \(\hat{y}_{i}\) matches the actual value \(y_{i}\). This can be the mean squared error (MSE) for regression or log loss for classification, and \(\Omega(f_{k})\) is the regularization term that penalizes the complexity of each individual tree \(f_{k}\), preventing overfitting. For trees, this term usually measures the number of leaves and the \(\ell_{2}\) norm of leaf weights as follows:  
\[\Omega(f)=\gamma T+\frac{1}{2}\lambda\sum_{j=1}^{T}w_{j}^{2},\]  
where \(T\) is the number of leaves in the tree, \(w_{j}\) are the weights of the leaves, and \(\gamma\) and \(\lambda\) are the hyperparameters that control the regularization. XGBoost is an additive model in which predictions are built iteratively. Starting with an initial prediction \(\hat{y}^{(0)}\), new trees are added to improve the predictions:  
\[\hat{y}^{(t)}=\hat{y}^{(t-1)}+f_{t}(x)\]where \(f_{t}(x)\) is the tree added at step \(t\), which aims to correct the errors from the previous step. Each tree attempts to minimize the residual errors from the previous iteration. The XGBoost algorithm iteratively uses gradient descent to minimize the objective function. Specifically, it computes the pseudo-residuals, which are the negative gradients of the loss function with respect to the current predictions:  
\[r_{i}^{(t)}=-\frac{\partial L(y_{i},\hat{y}_{i}^{(t-1)})}{\partial\hat{y}_{i}^ {(t-1)}}.\]  
These residuals \(r_{i}^{(t)}\) serve as the targets for the next tree to predict. The tree is fitted to minimize these residuals using a loss function, such as the MSE. XGBoost uses a second-order Taylor expansion of the loss function to optimize it effectively:  
\[L(y_{i},\hat{y}_{i}^{(t)})\approx L(y_{i},\hat{y}_{i}^{(t-1)})+g_{i}f_{t}(x_{ i})+\frac{1}{2}h_{i}f_{t}(x_{i})^{2},\]  
where \(g_{i}=\frac{\partial L(y_{i},\hat{y}_{i}^{(t-1)})}{\partial g_{i}^{(t-1)}}\) is the first-order gradient and \(h_{i}=\frac{\partial^{2}L(y_{i},\hat{y}_{i}^{(t-1)})}{\partial g_{i}^{(t-1) }}\) is the second-order derivative. The goal is to find a tree \(f_{t}(x)\) that minimizes this approximation of the loss. The algorithm builds decision trees by splitting the data at different points based on feature values. The gain \(G_{s}\) for each possible split was computed as follows:  
\[G_{s}=\frac{1}{2}\Bigg{[}\frac{G_{L}^{2}}{H_{L}+\lambda}+\frac{G_{R}^{2}}{H_{ R}+\lambda}-\frac{(G_{L}+G_{R})^{2}}{H_{L}+H_{R}+\lambda}\Bigg{]}-\gamma,\]  
where \(G_{L}\) and \(G_{R}\) are the sums of gradients for the left and right splits; \(H_{L}\) and \(H_{R}\) are the sums of Hessians for the left and right splits; \(\lambda\) is the regularization parameter; and \(\gamma\) is the penalty for each split, that is, regularization. Regularization is a key feature of XGBoost. It controls the complexity of the trees, prevents overfitting, and ensures that the model generalizes well. Regularization is achieved through the parameters \(\gamma\) to control the number of splits and \(\lambda\) to control the magnitude of the leaf weights. After training multiple trees, the final prediction for a sample was obtained by summing the predictions from all trees as follows:  
\[\hat{y}=\sum_{t=1}^{T}f_{t}(x),\]  
where \(f_{t}(x)\) denotes the prediction from the \(t\)-th tree.