## 5 Validation and Data Analysis with ML
### First Scenario: Experimentation without BCT
Here, numFolds (the specified number of folds of data used for pruning the decision tree) equals 3, minNum (minimum number of instances per leaf) equals 2 and minVarianceProp (the minimum proportion of the variance on all the data that need to be present at a node) equals 0.001. * In the WEKA environment, the SVM classifier is called SMO and it implements [PERSON]'s sequential minimal optimization algorithm for training a support vector classifier. For SMO, the following options are active: SMO -C 1.0 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K \"weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 25000\" -calibrator \"weka.classifiers.functions.Logistic -R 1.0E-8 -M -1-num-decimal-places 4\". Here, epsilon (the epsilon for round-off error) is set to 1.0E-12, SVM kernel (i.e., polynomial kernel) is set to -K \"weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 25000\" and calibrator (the calibration method to use) is set to \"weka.classifiers.functions.Logistic -R 1.0E-8 -M -1-num-decimal-places 4\". * The Random Forest classifier under WEKA was selected, which is defined as a class for constructing a forest of random trees with the following setting: RandomForest -P 100 -I 100 -num-slots 1 -K 0 -M 1.0 -V 0.001 -S 1, where bagSizePercent (size of each bag, as a percentage of the training set size) equals 100, numIterations (the number of trees in the random forest) equals 100 and maxDepth (the maximum depth of the tree, 0 for unlimited) is equal to 0. * Naive Bayes classifier, a statistical classifier, has been adopted. It assumes that the values of attributes in the classes are independent. This assumption is called class conditional independence and it is based on [PERSON] theorem. Under WEKA, the NB classifier is called Naive Bayes using estimator classes. Therefore, the Naive Bayes parameters were set as follows: useKernelEstimator (use a kernel estimator for numeric attributes rather than a normal distribution) is set to false, numDecimalPlaces (the number of decimal places to be used for the output of numbers in the model) is set to 2, and batchSize (the preferred number of instances to process if batch prediction is being performed) is set to 100. * Finally, AdaBoost classifier is used for comparison purposes. It is defined as an adaptive boosting algorithm based on minimizing the exponential loss function. Therefore, the AdaBoost parameters in WEKA were set as follows: AdaBoostM1 -P 100 -S 1 -I 10 -W -weka.classifiers.trees.DecisionStump, where the base classifier to be used is DecisionStump and batchSize is set to 100. Figure 8 represents the comparison of the Accuracy of studied classifiers performed under a WEKA data-mining environment. Figure 8: Accuracy percentage for studied classifiers. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{**Confusion Matrix**} & \multicolumn{2}{c}{**Target**} \\ \multicolumn{2}{c}{**Normal**} & \multicolumn{1}{c}{**Attack**} \\ \hline \multirow{3}{*}{Model} & Normal & True Positive & False Positive \\  & Normal & (TP) & (FP) \\ \cline{2-3}  & Attack & False Negative & True Negative \\ \cline{2-3}  & (FN) & (TN) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Confusion matrix. Figure 6: Dataset instances repartition used during training and test phases. Figure 7: ML metrics evaluation life cycle. As shown in Figure 8, the accuracy percentage is equal to 99.97%, 99.96%, 99.95%, 99.76% and 78.1% for artificial neural network (ANN), DT, SVM, RF and NB classifiers. ANN classifier outperforms Decision Tree, SVM, Random Forest and Naive Bayes classifiers in terms of accuracy percentage. Figure 9 presents the results obtained for the second and third metrics (Precision and Sensitivity) for the studied classifiers, considering the number of TP and FP. One can note that ANN, DT and SVM present the best performance in terms of TP, precision and sensitivity metrics. NB classifier shows the worst performance in terms of FP, precision and sensitivity metrics.