# Windowed Hamming Distance-Based Intrusion Detection for the CAN Bus
## 5 Performance Evaluation
### Experimental Results  
#### 5.3.1 Effect of Hyperparameters  
According to our proposed model, we need to discuss three hyperparameters. They are the number \(N\) of windows, the size \(K\) of the window, and the parameter \(\rho\). Parameters \(N\) and \(K\) determine the benchmark in our approach. We utilize this benchmark to establish and record the normal state of the vehicle. We take into account the fact that the vehicle's normal state is represented by at least one full cycle of all of its IDs. Hence, the selection of \(N\) and \(K\) aims to ensure that the benchmark includes the car ID for at least one complete cycle. During the analysis of the car-hacking dataset, we conduct a categorization of all messages into five distinct groups based on the number of messages between two identical messages. The results of this analysis are presented in Table 3.  
We observe that the interval between two occurrences of the same ID is consistently a multiple of 20. The minimum interval is 20 and the maximum interval is 2000. To ensure that our benchmark meets the required conditions, it is necessary for the benchmark to contain a minimum of 2000 frames. Therefore, when setting the value of \(K\) as the minimum interval of 20, the size of \(N\) should be at least 100 or greater. To enhance robustness, we increase the number of data frames included in the benchmark. For experimentation purposes, we select various parameter combinations of K and N, namely 40, 60, 80, and 100 for K, and 60, 80, 100, and 120 for N. We aim to identify the parameter combination that yields the best performance in terms of our evaluation metrics. As for the parameter \(\rho\), we choose \(\rho=3\) based on the three-sigma principle. We evaluate our optimal choice of \(N\) and \(K\) through the detection results of DoS and fuzzy attacks. The experimental result is shown in Figures 7 and 8. Figure 7 illustrates the results obtained under a DoS attack, while Figure 8 presents the results obtained under a fuzzy attack. As it is an anomaly detection model, we pay more attention to the recall value and the FS value. The reason is that for the hacker's attack on the vehicle, the harm of not detecting the attack frame is far greater than the false alarm of the normal frame. We can sacrifice a certain amount of precision under the condition of pursuing high recall. In Figure 7b, we observe a higher recall value for detecting DoS attacks when the value of \(K\) is set to 60. On the other hand, in Figure 8b, a higher F1-score (FS) is achieved in the detection of fuzzy attacks when the value of \(K\) is set to 80. When the value of \(K\) is set to 80, we observe a decrease in recall for detecting DoS attacks. However, the FS shows only slight changes. Based on these observations, we believe that setting \(K\) to 80 would result in good performance for our method. We speculate that the possible reason for the decrease in recall is that in DoS attacks, as K increases, more attack data are required to decrease the Hamming distance below threshold. During the initial stages of an attack, the number of DoS frames injected may be insufficient to be detected, resulting in weaker detection capabilities during this period. The fuzzy attack demonstrates better resistance to this flaw, possibly due to its random nature. As shown in Figures 7a and 8a, it is shown that for detecting both DoS attacks and fuzzy attacks, the impact of \(N\) on the method's performance is lesser when compared to \(K\). This result could be attributed to the primary role of \(N\), which is to enhance the method's robustness. It can be observed that when the \(N\) is 100, the method exhibits a favorable performance in detecting both types of attacks. \begin{table}
\begin{tabular}{l l l} \hline \hline
**ID Type** & **Interval** & **Number** \\ \hline A & 20 & 17 \\ B & 40 & 4 \\ C & 100 & 1 \\ D & 200 & 2 \\ E & 2000 & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Analysis of id intervals.