# Windowed Hamming Distance-Based Intrusion Detection for the CAN Bus
## 5 Performance Evaluation
### Comparison with Other Work  
In Table 5, we compare the experimental results of our method with those of other existing methods. To ensure a more scientifically rigorous comparison, all the methods chosen for evaluation are conducted on the car-hacking dataset. We select the following methods for comparison: the histogram-based method proposed by [PERSON] et al. [13], the support vector classifier (SVC) method introduced by [PERSON] et al. [53], the DCNN-based method developed by [PERSON] et al. [9], the method utilizing the Hamming distance based on ID presented by [PERSON] et al. [12], and the Ensemble Learning-Based method, which is the latest approach proposed by [PERSON] et al. [34]. These selected methods encompass a variety of techniques, including statistical methods, traditional machine learning methods, and neural network methods. Notably, the histogram-based method among the traditional machine learning approaches utilizes the sliding window method. Thus, our chosen comparison approach offers a comprehensive assessment by considering a diverse range of methodologies.  
Compared to the DCNN [9] method, our model demonstrates a slight disadvantage, with a maximum reduction in FS of 1.78% and a minimum reduction of 0.34%. However, deep learning methods often increase detection time and memory usage to achieve better performance. To meet real-time monitoring standards, slight performance sacrifices may be accepted. In the subsequent section, we compare the detection time and memory usage between our method and the DCNN method. In comparison to the method utilizing SVC [53], our approach improves by 0.91% in the FS. Moreover, the method [12] solely relying on Hamming distance achieves a minimum accuracy of only 20% to 30% in detecting fuzzy attacks. In contrast, our method, employing a windowing approach, elevates the accuracy of fuzzy attack detection to 98.66%. The histogram-based method [13] also employs the sliding window method. While both methods achieve a perfect 100% F1-Score (FS) against both types of spoofing attacks, our method excels in detecting DoS and fuzzy attacks, showcasing an increased accuracy of 3.49%. Finally, we compare it with the latest method [34], which uses ensemble learning-based and multi-classification model anomaly detection. Judging from the experimental results, our experimental results are significantly ahead.  
The above comparison fully demonstrates the stability and superiority of our model, which has a stable effect on any attack type and achieves outstanding results.  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Model** & **Type** & **Accuracy** & **Precision** & **Recall** & **FS** \\ \hline \multirow{4}{*}{DCNN [9]} & DoS & 99.97\% & 100\% & 99.89\% & 99.95\% \\  & Fuzzy & 99.82\% & 99.95\% & 99.65\% & 99.80\% \\  & Spoofing Gear & 99.95\% & 99.99\% & 99.89\% & 99.94\% \\  & Spoofing Rpm & 99.97\% & 99.99\% & 99.94\% & 99.96\% \\ \hline SVC(MGA) [53] & Multiclasss & 98.7\% & 98.7\% & 98.7\% & 98.7\% \\ \hline Stacking [34] & Multiclasss & 98.5\% & 98.7\% & 98.5\% & 98.5\% \\ \hline \multirow{4}{*}{H-IDFS [13]} & OCSVM-DoS & 97.28\% & 100\% & 96.20\% & 98.06\% \\  & OCSVM-Fuzzy & 95.17\% & 99.55\% & 94.93\% & 97.18\% \\  & OCSVM-Gear & 100\% & 100\% & 100\% & 100\% \\  & OCSVM-RPM & 100\% & 100\% & 100\% & 100\% \\ \hline \multirow{4}{*}{Hamming distance [12]} & Fuzzy-NoRange & 98\% & - & - & - \\  & Fuzzy-SmallRange & 96\% & - & - & - \\  & Fuzzy-MidRange & 20–30\% & - & - & - \\ \hline \multirow{4}{*}{Ours} & DoS & 99.76\% & 99.91\% & 99.31\% & 99.61\% \\  & Fuzzy & 98.66\% & 99.68\% & 96.38\% & 98.02\% \\ \cline{1-1}  & Spoofing Gear & 99.48\% & 99.89\% & 98.66\% & 99.27\% \\ \cline{1-1}  & Spoofing Rpm & 99.61\% & 99.99\% & 98.98\% & 99.48\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison with other works employing car hacking.