# XGB-RF: A Hybrid Machine Learning Approach for IoT Intrusion Detection
## 2 Materials and Methods
### Classifiers
#### 2.4.1 Random Forest (RF)  
As the name suggests, the random forest (RF) algorithm is a collection of classification trees where each tree casts one vote for the most frequent class being assigned to the input data [30]. Afterward, a class with majority votes gets selected as the outcome. [PERSON] in 2001, introduced the idea on top of the Bagging (Bootstrap-Aggregating) model, which is currently known as random forest. Not only for classification and regression, but RF is also being employed for feature selection (FS) purposes. [PERSON] et al., in their work, contributed with the methodological model to rank variable importance through depth analysis on variable importance index [31]. Sensitivity of sample size, variable quantity, responsiveness towards different method parameters, and sensitivity towards presence of correlated variables are observed to identify RF variable importance score in their work. Being a supervised learning algorithm, random forest forms the forest, which is a part of a decision tree. It outperforms many machine learning algorithms in terms of accuracy, particularly for big datasets with numerous features. It is usually trained in the bagging method in which a randomized selection (without replacement) is performed from the training samples to construct each tree. It implies that the new trees are not reliant on older ones. Depending on the predictions of decision trees, the model aggregates its output. In general, few measurements are applied to calculate feature importance such as Gini index, mean decrease accuracy, permutation importance method (overcomes the imperfection of mean decrease impurity), etc. Due to the use of the bagging model, RF has gained significance in reducing data over-fitting phenomenon and variance among decision trees. Furthermore, handling missing data are a common treatment of RF architecture. The RF classifier is integrated with the number of classification trees. Based on the feature importance criteria, 40 notable features are selected. The classification result is calculated using Equation (1). \[C(t)=\max_{P}E_{t}\sum_{i=1}^{K}(c_{i}(T)=P) \tag{1}\]  
where \(T\) is the training set from the original dataset \(S\) and \(K\) indicates the subsets from the \(T\) dataset. For each subset, the algorithm automatically generates \(K\) decision trees with the help of a random vector. \(C(t)\) represents the classification result where \(c_{i}(T)\) denotes the classification result of \(i^{th}\) decision tree. Here, \(P\) is the target category. However, several random forest hyper-parameters are being used to either enhance the model's prediction capability or speed up the algorithm. RF gains higher-level performance in the case of high-dimensional data by completing an implicit FS process. Gini importance can be used within RF as measurement criteria for finding feature importance. These relevance scores help identify the decision trees significant to the classifier and can be considered as an outgrowth. \[i(t)=1-f_{1}^{2}-f_{0}^{2} \tag{2}\]Equation (2) is used to measure the Gini impurity. Here, each node is represented as \(t\), which can be any node of RF decision trees. Gini impurity is used to find the optimum split, which is an estimation of measuring entropy. Furthermore, \(f_{j}\) in Equation (3) is the fraction of \(n_{j}\) samples out of the total number of samples \(n\) and \(j\) = 0,1 represents the class. \[f_{j}=\frac{n_{j}}{n} \tag{3}\]  
We can achieve decreasing \(\delta i\) by splitting and conveying products to two separate sub-notes (\(t_{p}\) & \(t_{q}\)) by a threshold on variable \(\Theta\). Equation (4) reflects the procedure. \[\delta i(t)=i(t)-f_{p}i(t_{p})-f_{q}i(t_{q}) \tag{4}\]  
Next, an exhaustive search is performed with all-inclusive values of \(\Theta\), which is obtainable in the node overall thresholds. Afterward, considering all nodes \(t\), Gini impurity values' reductions are saved for all variables separately with Equation (5).