# Connected Vehicles Security: A Lightweight Machine Learning Model to Detect VANET Attacks
## 5 Results and Discussion
Table 11 shows the accuracy and processing time. It can be noted that, for each classifier, while the accuracy remains the same, the training time, testing time, and execution time are decreased, which means there is improvement in terms of processing cost due to the application of the information gain feature selection method. Figure 2: The computer processing time for the AdaBoost, RF, MLP, DT, and KNN classifiers. The second layer of enhancement is the development of a balanced dataset that can reduce bias in the prediction process and improve classification accuracy. As described in the Method section, a random oversampling method was implemented using Python software for the CICIDS2017-GI dataset, which was developed by applying the gain information feature selection method. For this study, the developed balanced dataset was called Balanced CICIDS2017 with gain information (BCIC2017-GI). All classifiers were trained, tested, and evaluated using the BCICIDS2017-GI dataset which includes 1,700,330. A ten-fold cross-validation technique was employed for each model to validate the results. The computed evaluation metrics, which include accuracy, recall, precision, F1 score, the Matthews correlation coefficient (MCC), and the area under the ROC curve (AUC), are listed in Table 12.  
The results show a significant improvement in the classification accuracy obtained by all classifiers; however, RF was dominant. It achieved an outstanding accuracy rate of 99.8%, followed by AdaBoost, and MLP. DT and KNN are at the bottom level with accuracies of 98.5% and 98.40%, respectively. Figure 3 shows the accuracy rates achieved by the benchmark classifiers. When examining the accuracy for each attack among all these classifiers, Table 13 shows good performance for all classifiers after using the developed dataset, BCICIDS2017-GI. These results support the conclusion that RF is the best. It demonstrated the best performance with a high capacity to detect all attacks, with an outstanding classification accuracy rate. \begin{table}
\begin{tabular}{c c c c c c c} \hline  & **Botnet** & **Brute** & **Dos/Dos** & **Infiltration** & **PortScan** & **Web** & **Attack** & **Normal** \\ \hline DT & 99.5 & 99.9 & 99.6 & 100 & 99.8 & 99.6 & 98.7 \\ \hline RF & 100 & 100 & 99.8 & 100 & 100 & 99.8 \\ \hline AdaBoost & 100 & 100 & 99.7 & 100 & 100 & 100 & 99.7 \\ \hline MLP & 100 & 99.7 & 99.6 & 100 & 99.8 & 99.7 & 99.3 \\ \hline KNN & 99.9 & 99.9 & 99.0 & 100 & 99.7 & 99.9 & 98.6 \\ \hline \end{tabular}
\end{table} 
Table 13: Attack Classification Accuracy. \begin{table}
\begin{tabular}{c c c c c c} \hline  & **Accuracy** & **Training Time (s)** & **Testing Time (s)** & **Execution Time (s)** \\ \hline MLP & 98.9 & 2743.669 & 2.114 & 2745.783 \\ \hline RF & 99.5 & 36.345 & 1.656 & 38.001 \\ \hline AdaBoost & 99.3 & 53.566 & 1.091 & 54.657 \\ \hline DT & 97.9 & 63.832 & 0.060 & 63.892 \\ \hline KNN & 96.9 & 6.839 & 102.980 & 109.819 \\ \hline \end{tabular}
\end{table} 
Table 11: Accuracy and Processing Time for All Classifiers. It is noteworthy that AdaBoost comes next, and demonstrates excellent performance. However, RF still surpassed all of them when considering processing time and cost, as is shown in Figure 2. Figure 4 shows the accuracy achieved by all classifiers for each attack which clearly indicates that our proposed classifier (RF) is a dominant classifier. For further evaluation, the ROC curves for DT, AdaBoost, MLP, KNN, and the suggested classifier (RF) are shown in Figure 5, Figure 6, Figure 7, Figure 8, and Figure 9, respectively. The ROC curves and performance metrics showed how well each classifier performed, indicating that the proposed classifier performed better than other classification techniques. On the other hand, the confidence interval (CI) for the proposed classifier is 0.034555021 with 99.83455502 and 99.76544498 as the upper bond and lower bond, respectively.