# Connected Vehicles Security: A Lightweight Machine Learning Model to Detect VANET Attacks
## 5 Results and Discussion  
The selected classification models, including RF, AdaBoost, DT, KNN, and MLP were built, trained, and tested using the revised version of the CICIDS2017 dataset. The computed accuracy, recall, precision, F1 score, and Matthews correlation coefficient (MCC) are listed in Table 9.  
The performance metrics indicate that the RF classifier outperforms all the other benchmark classifiers. It achieved 99.5% for all performance metrics: accuracy, recall, precision, and F1 score. For visualization, all of the computed metrics are shown in Figure 1.  
Another important factor supporting this finding is the computational cost of each model. To assess this, the processing time for all classifiers was computed, and this indicates that RF still outperforms the other classifiers by consuming the least processing time. Figure 2 shows the processing times for all classifiers while Table 10 lists the training time, testing time, and computation time. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Classifier** & **Training Time (s)** & **Testing Time (s)** & **Computation Time (s)** \\ \hline KNN & 11.374 & 138.747 & 150.121 \\ \hline MLP & 3188.379 & 2.548 & 3190.927 \\ \hline RF & 38.043 & 1.827 & 39.87 \\ \hline DT & 78.155 & 0.053 & 78.208 \\ \hline ADA & 55.246 & 1.406 & 56.652 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Execution Time for All Classifiers. Figure 1: The accuracy of the MLP, RF, AdaBoost, DT, and KNN classifiers. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Accuracy** & **Recall** & **Precision** & **F1 Score** & **MCC** \\ \hline MLP & 98.9 & 98.9 & 98.9 & 98.9 & 98.5 \\ \hline RF & 99.5 & 99.5 & 99.5 & 99.5 & 99.3 \\ \hline AdaBoost & 99.3 & 99.3 & 99.3 & 99.3 & 99.0 \\ \hline DT & 97.9 & 97.9 & 97.9 & 97.8 & 97.2 \\ \hline KNN & 96.9 & 96.9 & 96.9 & 96.8 & 95.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Performance Metrics for All Classifiers when using the Revised Version of the CICIDS2017 Dataset. It can clearly be noted that RF also performed significantly better than the other classifiers, followed by AdaBoost, DT, and KNN, while MLP lags behind by consuming too much processing time, which is normal because DL methods are computationally expensive. The computational cost is a crucial factor when considering networks that require real-time actions. This confirms that the RF classifier is much better; it requires only 71%, 51%, 27%, and 1.3% of the Ada-Boost, DT, KNN, and MLP processing times, respectively. After concluding that RF was the best classification model in terms of accuracy and processing cost, two layers of enhancements were applied to further improve both the classification accuracy and processing cost. These two layers include (i) applying an appropriate feature selection approach to only choose the features with a high level of influence on the classification results, which boosts both the prediction accuracy and processing cost, and (ii) applying a resampling method to solve the imbalance problem in the dataset. Having a balanced dataset can significantly enhance prediction performance and eliminate bias in the classification results. In the first layer of enhancement, the information gain feature selection method described in the Method section was applied. Accordingly, 61 features with a score greater than 0.1 were selected to form the newly developed dataset, CICIDS2017 with gain information (CICIDS2017-GI). All classifiers were trained on the aforementioned dataset, CICIDS2017-GI. Almost all classifiers achieved the same classification accuracies, which were achieved before applying the feature selection method. This indicates that this feature selection technique successively selects the effective features that have a significant influence on prediction accuracy and neglects other features that do not. In addition, removing features that have no weight in the classification can significantly reduce the processing cost, given that the number of features to be processed is reduced.