## 3 Materials and Methods
### Methodology
#### 3.2.2 Modeling with XGBoost and Hyperparameter Optimization  
In this study, the XGBoost method [9] was used as the supervised classification model. The method originated from gradient tree boosting algorithm [44; 45], which is an effective ML method. It uses regularized boosting technique to reduce the overfitting and thus increase the model accuracy. XGBoost offers scalability for different scenarios, handling of sparse data, availability of comprehensive documentation, low computational resource requirement and high performance (i.e., speed), and the ease of implementation [9]. The method was selected here for being the winner of many data science competitions [9; 46]. Further modifications to the method for highly imbalanced datasets also exist (e.g., [47]). It was implemented using the XGBoost Python package [9] in Python 3.7 environment in this study. XGBoost is an optimized extension of the gradient boosting algorithm. The main idea of a boosting algorithm is combining weak learners outputs sequentially to achieve better performance [48]. XGBoost uses many classification and regression trees (CART) and integrates them using the gradient boosting method. XGBoost has three important aspects, which are regularized objective function for better generalization, gradient tree boosting for additive training, and shrinkage and column subsampling for preventing overfitting [9]. The Equations (3)-(6) summarized from [9] depict the algorithmic process. The XGBoost algorithm aims at minimizing the regularized objective function given in Equation (3). \[\mathrm{L}(\Phi)=\sum_{i}\mathrm{I}(\hat{y}_{i},y_{i})+\sum_{k}\Omega(\mathrm{ f}_{k}) \tag{3}\]  
The first term in Equation (3) is a loss function, which measures the difference between the target \(y_{i}\) and the prediction \(\hat{y}_{i}\), and the second one is a penalty term as explained in Equation (4) for controlling model complexity to avoid overfitting. \[\Omega(\mathrm{f})=\ \gamma\mathrm{T}\ +\frac{1}{2}\lambda||w||^{2} \tag{4}\]  
where;  
T: the number of leaves in the tree;w: the score of each leaf;  
\(\gamma,\lambda\) : the regularization degrees. An iterative approach is applied in the method to minimize the objective function. The objective function to be minimized at step t is given in Equation (5). \[\mathrm{L^{(t)}}=\sum_{i=1}^{n}\mathrm{I}(\mathrm{y_{i}}\mathrm{\hat{y}_{i}^{(t -1)}}+\ \mathrm{f_{t}(x_{i})})+\ \mathrm{\Omega(f_{t})} \tag{5}\]  
In order to speed up the optimization process, second order Taylor expansion is applied to the objective. After removing the constant terms, a simplified objective function at step t is given in Equation (6). \[\mathrm{\widetilde{L}^{(t)}}=\sum_{i=1}^{n}[\mathrm{g_{i}f_{t}(x_{i})+\frac{ 1}{2}h_{i}f^{2}(x_{i})}]+\ \mathrm{\Omega(f_{t})} \tag{6}\]  
where;  
\[\mathrm{g_{i}}=\partial_{\mathrm{\hat{y}^{(t-1)}}}\mathrm{I}(\mathrm{y_{i}} \mathrm{\hat{y}^{(t-1)}})\mathrm{h_{i}}=\partial_{\mathrm{\hat{y}^{(t-1)}}}^{ 2}\mathrm{I}(\mathrm{y_{i}}\mathrm{\hat{y}^{(t-1)}}) \tag{7}\]  
In addition to the regularization, shrinkage and feature subsampling are used to prevent overfitting in the XGBoost. There are several algorithms, such as basic exact greedy, approximate, weighted quantile sketch and sparsity-aware split finding, which are available in XGBoost to be used in finding the best split efficiently. The training dataset prepared in the previous stage was split as training (80%) and testing (20%) sets. The training subset was again divided as training (90%) and validation (10%) sets for the purpose of hyperparameter optimization of the model. The testing dataset (20%) was used for the quality assessment only after the final model was constructed and trained. Although there is no exact criteria or rule of thumb on the ratio selection for the dataset splitting, 70/30 and 80/20 are commonly observed ratios in the literature. The dataset size is an important criterion for this purpose. Since the study dataset was relatively large, the 80/20 approach was followed here and no over- or underfitting was experienced.