## 5 Validation and Data Analysis with ML
### First Scenario: Experimentation without BCT
As shown in Table 7, the confusion matrix is a table showing the number of instances belonging to each of four categories (represented by TP, FP, FN and TN). The ML performance metrics used in this study are as follows:* Accuracy defined as: \[\text{Accuracy}=\frac{\left(\text{TP }+\text{ TN}\right)}{\left(\text{TP }+\text{ TN }+\text{ FP }+\text{ FN}\right)}\times 100\] (1)
* Precision defined as: \[\text{Precision}=\frac{\text{TP}}{\left(\text{TP }+\text{ FP}\right)}\times 100\] (2)
* Sensitivity defined as the proportion of actual positives which are predicted positive: \[\text{Sensitivity}=\frac{\text{TP}}{\left(\text{TP }+\text{ FN}\right)}\times 100\] (3)
* Matthews Correlation Coefficient (MCC): defines the correlation between the predicted value and the observed one [4]. MCC measures the quality of a classifier to perform a classification task. \[\text{MCC}=\frac{\text{TP}\times\text{TN}-\text{FP}\times\text{FN}}{\sqrt[ 2]{\left(\text{TP}+\text{TN}\right)\times\left(\text{TP}+\text{FN}\right) \times\left(\text{TN}+\text{FP}\right)\times\left(\text{FP}+\text{FN}\right)}} \times 100\] (4)  
The different required steps to perform the various ML metrics are depicted in Figure 7. Therefore, the life cycle of ML evaluation metrics involves six steps: dataset selection, preprocessing, classifier model selection, model training, test phase and ML evaluation metrics. In the pre-processing step, a correlation-based feature selection (CFS) is employed with the best first search approach. Indeed, the CFS algorithm starts with an empty node and can go up to five nodes. In addition, subsets are evaluated with 10-fold cross-validation on the training dataset. For instance, the different steps needed for simulation experiments to carry out predictive accuracy for the ANN classifier are defined as follows:  
1. Upload the TON_IoT dataset. 2. Data pre-processing (a CFS is selected for the attribute evaluator and best first is selected for the search method). 3. Select the classifier type and configure the classifier parameters. 4. Model training. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **ANN** & **DT** & **SVM** & **RF** & **NB** & **AdaBoost** \\ \hline Time to build the model (s) & 3232.14 & 30.59 & 2.31 & 13.6 & 0.55 & 3.81 \\ Time to test model on training data (s) & 2.72 & 0.12 & 0.22 & 0.42 & 3.06 & 0.55 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Required time for training and testing phases for the studied classifiers. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Type** & **Label** & **Description** \\ \hline
0 & normal & Normal activity \\ \hline
1 & ddos & Distributed denial of service attack \\ \hline
2 & dos & Denial of service attack \\ \hline
3 & injection & Injection attack \\ \hline
4 & mitm & Man in the middle attack \\ \hline
5 & password & Brute force attack \\ \hline
6 & xss & Cross-site scripting attack \\ \hline
7 & scanning & Port scanning attack \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Type and label attributes for attack detection. Test phase and select the test options (cross-validation = 10-fold). 6. Run the simulation to evaluate predictive accuracy. The parameters set on the WEKA environment for the different used classifiers in the experiments are the following. * The name of the ANN classifier in the WEKA environment is called multilayer perceptron: a classifier using the backpropagation technique to classify instances. In our experiments, the multilayer perceptron classifier parameters were set as follows: MultilayerPerceptron -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H a. Here, the LearningRate (the learning rate for weight updates) is equal to 0.3, Momentum = 0.2, TrainingTime = 500, Validation threshold = 20, HiddenLayer (to define the hidden layers of the neural network) equal to a (attributes + classes) * Regarding the decision tree classifier under the WEKA environment, REPTree classifier was selected and defined as a fast decision tree learner with the following setting: REPTree -M 2 -V 0.001 -N 3 -S 1 -L -1 -I 0.0.