## 4 Results and Discussion  
Evaluation of the trained models on the hardware PASTA testbed (see Figure 2) used a single drive cycle consisting of three 30 s periods of acceleration to, and holding of certain speeds (i.e., 50 kph, 100 kph, and 199 kph) separated by 10 s stopped intervals. Since we use vehicle speed as our performance parameter, attack messages consist of injection attacks targeting the accelerator and brake pedal inputs from the driver. The attack consists of simultaneously injecting 0x01A messages (brake_operation_indicator) with a value of 100% and 0x02F messages (accelerator_pedal_operation_indicator) with a value of 0% at 5 ms intervals. We perform ~15 runs with each type of defense and average the results to obtain the final QMoCR measurement. For each learning approach, five independent models were initialized and trained, and their evaluation results were averaged. The performance of each defense is visualized in Figures 5 and 6, and average performance measurements are provided in Table 4. While we do not have sufficient samples to quantify the correlation between traditional evaluation measurements and real-world performance, as measured by QMoCR and depicted in Figure 6, the relationship does not appear to scale linearly. Although all classifiers achieve accuracy above ~70%, the timing of allowed attack messages (i.e., false negatives) in relation to legitimate control messages (i.e., true negatives) is also important to overall system performance. Without defense, the vehicle is undrivable when under attack (Figure 6a). All defenses performed in line with their accuracy with regard to vehicle speed when not subjected to an attack (not illustrated), resulting in no distinguishable degradation in measured performance over the baseline run for defenses with accuracy above 90%. This is notable because there is potential for impact on the vehicle caused simply by the addition of the defense without any attack present. In fact, there would be noticeable degradation had we focused on vehicle functionality enabled by brief changes in CAN bus message data, such as on/off activation messages. In those cases, large impacts to vehicle functionality can occur even with few but critically timed false positives that block the intended control signal. \begin{table}
\begin{tabular}{|c|c c|c c c c c c|} \hline \multirow{2}{*}{**Defense**} & \multicolumn{3}{c|}{**QMoCR**} & \multicolumn{6}{c|}{**Traditional Performance Measures**} \\ \cline{2-9}  & **AUC** & **Defense/Baseline** & **AUC Ratio** & **Accuracy** & **Precision** & **Recall** & **F-Score** & **Sensitivity** & **Specificity** \\ \hline Baseline & 9615 &... &... &... &... &... &... &... \\ \hline No Defense & 4 & 0\% &... &... &... &... &... &... \\ \hline Static Timing & 3976 & 41\% & 0.9325 & 0.9251 & 0.9300 & 0.9275 & 0.9300 & 0.9346 \\ \hline RL Model & 8606 & 90\% & 0.9341 & 0.943 & 0.913 & 0.9277 & 0.913 & 0.9523 \\ \hline Static Timing w/Frequency & 9430 & 98\% & 0.9433 & 0.9345 & 0.9443 & 0.9393 & 0.9443 & 0.9425 \\ \hline CNN Model & 109 & 1\% & 0.7009 & 0.7592 & 0.518 & 0.6158 & 0.518 & 0.8585 \\ \hline LSTM Model & 0 & 0\% & 0.6995 & 0.7697 & 0.4998 & 0.6061 & 0.4998 & 0.8714 \\ \hline \end{tabular}
\end{table} 
Table 4: Performance measurements of defenses using a drive cycle on the PASTA testbed. Figure 5: Plots of the multiple evaluations run on PASTA hardware per defense while under attack. LSTM is not shown as it was unable to complete a drive cycle due to blocking transmission shifting messages. Colored lines represent unique trials and are plotted to show the variability of each method. The RL model (**c**) appears as reliable as the hand-crafted oracle method (**d**). The RL model (Figures 5c and 6d) performed better than the static timing defense alone (Figures 5b and 6c). This result is expected because the RL model adapts during training to recognize variations in message timing, order, and data content, whereas the static timing defense allows all traffic in the allowed time window, whether it is malicious or not.