# Windowed Hamming Distance-Based Intrusion Detection for the CAN Bus
## 5 Performance Evaluation
### Experimental Results
#### 5.3.2 Result in Two Datasets  
First, we experiment with our model on car hacking [14]. Figure 10 shows the results of four different attack types in the form of a confusion matrix. Table 4 presents the four evaluation criteria we selected in a tabular form. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Dataset** & **Attack Types** & **Accuracy** & **Precision** & **Recall** & **F-score** \\ \hline \multirow{4}{*}{Car Hacking} & DoS & 99.76\% & 99.91\% & 99.31\% & 99.61\% \\ \cline{2-6}  & Fuzzy & 98.66\% & 99.68\% & 96.38\% & 98.02\% \\ \cline{2-6}  & Spoofing Gear & 99.48\% & 99.89\% & 98.66\% & 99.27\% \\ \cline{2-6}  & Spoofing Rpm & 99.61\% & 99.99\% & 98.98\% & 99.48\% \\ \hline Xiaoyu & DoS & 99.60\% & 99.65\% & 99.45\% & 99.55\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Result of model under two datasets. Figure 10: All attack types of confusion matrix result in car hacking. Figure 9: The influence of hyperparameter \(\rho\) on experimental results. Among the four evaluation metrics, accuracy reflects the method's overall ability to accurately detect anomalies. Precision represents the method's capability to correctly identify normal data, while recall measures the method's effectiveness in detecting attacks. The FS provides a comprehensive assessment of the model's performance by considering the detection of both normal data and attacks, thus capturing the overall effectiveness of the method in anomaly detection. In intrusion detection tasks, particularly in the context of automotive systems, the impact of a missed attack can be significant. Additionally, while the method has the ability to detect attacks, it is equally important to minimize false positives. Hence, in the evaluation of the method's performance, greater attention is placed on recall and FS. In Table 4, it is evident that the method exhibits a robust detection capability for DoS attacks, as indicated by the high FS of 99.61%. However, the detection performance for fuzzy attacks is relatively weaker, with an FS of only 98.02%, primarily due to a lower recall rate. Fuzzy attacks employ randomly injecting data frames, resulting in non-fixed patterns for the injected frames. Randomly injected frames within the window cause an increased Hamming distance from the normal state, surpassing the predefined threshold. The method leverages this characteristic to detect fuzzy attacks. However, due to the nature of random injection, partially injected data frames may not have enough impact to increase the Hamming distance above the threshold. As a result, the method exhibits a weakness in detecting fuzzy attacks. However, other types of attacks, such as DoS attacks, exhibit more fixed patterns in their data frames. These fixed patterns result in consistent changes to the Hamming distance, allowing for better detection results. Overall, the method demonstrates high efficiency in detecting the four types of attacks, with FS exceeding 98%. The recall achieves the highest value of 99.31%. Precision, on the other hand, achieves the lowest value of 99.68%. Accuracy ranges from a minimum of 98.66% to a maximum of 99.76%. In Figure 11, we present the confusion matrix of the method applied to the Xiaoyu dataset. The evaluation metrics for the method are displayed in the last row of Table 4. The Xiaoyu dataset solely consists of normal data. Therefore, to simulate DoS attacks, we introduce a substantial volume of high-priority data into the Xiaoyu dataset over a specific time period. As indicated in Table 4, the method achieves impressive results on the Xiaoyu dataset, with accuracy of 99.60%, precision of 99.65%, recall of 99.45%, and F1-score (FS) of 99.55%. These results demonstrate the method's strong effectiveness in detecting DoS attacks. Comparing the results between car hacking and the Xiaoyu dataset, we observe a slight decrease of 0.06% in the FS for the Xiaoyu dataset.