# XGB-RF: A Hybrid Machine Learning Approach for IoT Intrusion Detection
## 2 Materials and Methods
### Classifiers
#### 2.4.2 eXtreme Gradient Boosting (XGBoost)  
After selecting features using RF, an eXtreme Gradient Boosting classifier is utilized in the proposed approach to detect botnet attacks based on a few selected preconditions. XGBoost, also known as XGB, is a promising tree-based ensemble learning classifier, which is treated as the most effective implementation of gradient-boosted decision trees. Gradient boosted decision trees utilize a series of decision trees, each of which learns from the preceding tree and affects the subsequent tree; therefore, they improve the model and develop a powerful learner [32]. XGBoost combines weak classifiers to create a strong one [33]. As opposed to RF, where independent trees are generated and work individually, XGBoost incorporates feedback from previously accepted decision trees. Each iteration of gradient boosting optimizes the given loss function. The goal is to minimize the residual from the previous step where residual can be interpreted as the dissimilarity between predicted estimation and true estimation. Upon reaching a threshold point for the residual value, the final model is declared for further use. Nonetheless, if a number of decision trees fall under a threshold value before the residual can drop under the threshold, training is stopped and the final model is selected. The use of parallel execution, faster execution than gradient boosting, acceptance of regularization, etc., are a few salient features of the XGB model. XGBoost's objective function [34] for evaluating the model's performance can be represented as Equation (6). \[P(\theta)=t(\theta)+r(\theta) \tag{6}\]  
where \(\theta\) represents the parameters, \(r\) denotes the term of regularization, and \(t\) is the loss of training. Algorithm 2 depicts the XGBoost model architecture. ```
1:Input: \(\theta\), \(\theta_{1}\), \(\theta_{2}\), \(\theta_{3}\), \(\theta_{4}\), \(\theta_{5}\), \(\theta_{6}\), \(\theta_{7}\), \(\theta_{8}\), \(\theta_{9}\), \(\theta_{10}\), \(\theta_{11}\), \(\theta_{12}\), \(\theta_{13}\),


```
Input:\(S\in\mathcal{R}^{n\times d}\) where data are of \(d\) dimension and n represents number of samples. Target:\(F\in\mathcal{R}^{n\times 1}\) Output:\(\sum_{i=1}^{C}P_{i}=1,\forall i\in C=2\) where \(C\) = the number of classes and \(P\in[0,\,1]\) for unrevealed test set \(s\). \(P\) represents posterior probability
1 Initialization:\(F_{o}(s)=\operatorname*{argmin}_{\Omega}\sum_{i=1}^{T}L(Y,\Omega)\)[33], where \(L(Y,\,F(s))\) represents differentiable loss function and \(T\) is the total number of sampling. for\(k=1\) to \(M\left(No.\,of\,\,iterations=n\right)\)do
2 Compute pseudo-residuals, \(pr_{im}=-[\frac{\delta L(F,F(S_{i}))}{\delta F(S_{i})}]\), where \(i=1,2,...,N\)
3 Fit a base tree, \(h_{m}\) deploying training set \((S_{i},pr_{im})\) where \(i=1,2,...,N\)
4 Computing the multiplier \(\Omega_{k}\) by \(\Omega_{k}=\operatorname*{argmin}_{\Omega}\sum_{i=1}^{n}L(F_{i},f_{k-1}(S_{i} )+\gamma h_{k}(S_{i}))\)
5 Upgradation of model \(f_{k}(s)=f_{k-1}(s)+\Omega_{k}h_{k}(s)\)
6 end for
7\(f_{k}(s)\) is the target posterior probability, \(P\in[0,\,1]\)
```

**Algorithm 2**Followed steps for XGBoost classifier

## 3 Experimental Results

The experimental results are based on the original test data. Hold out approach is used for dividing the dataset. Here, 75% of data are used for training, and the remaining 25% are used for testing purposes. The performance comparison table among classification algorithms and the confusion matrices are shown in Sections 3.1 and 3.2. In addition, evaluation on different Train-Test schemes, Receiver Operating Characteristic (ROC) Curve and comparison with other studies are demonstrated in Sections 3.3-3.5, respectively. ### Performance Measures

After selecting features with RF and classifying these using XGB, some statistical measures are performed to assess the performance of the proposed method.