## 4 Results and Discussion
If a malicious accelerator or brake pedal message occurs after the legitimate message and is still within the window, which is not uncommon, it will impact the vehicle's performance. Also, as expected, the RL model performed worse than the customized static timing defense with frequency analysis (Figures 5d and 6e). When a defense is custom-crafted for a malicious accelerator or brake pedal message, the RL model is able to recognize the victim's  
particular scenario, even as simplistic as this one is, it can perform very well under those circumstances. The issue with this custom-crafted approach is twofold. First, it takes time, expertise, and awareness to implement custom defenses for every scenario. Second, this custom defense performs poorly when the attack changes (e.g., injecting random message values or time syncing the attack messages to arrive right before the legitimate messages). The RL model is not as sensitive to such changes as it would have experienced similar situations in training. The CNN (Figures 5a and 6b) and LSTM (not illustrated) models performed worse than the RL model during evaluation. This is exemplified in the QMoCR measurement, where RL averaged 90% while CNN averaged 1%. The LSTM model failed to complete a drive cycle due to blocking legitimate transmission shifting messages, which kept the testbed from shifting into the drive. This analysis also points out how the QMoCR measurement can benefit from the additional context provided by the AUC graphs. While the overall number may be sufficient for comparison during true tests of a system's ability to meet maximum KPPs, it does not solely provide all relevant information about changes in cyber resilience, which may be important when viewed in the context of a vehicle performing a mission. For example, one could imagine two defenses with identical QMoCR measurements; however, one might have many small performance degradations with gradual recovery, while the other may have a few larger performance degradations with faster recovery. These and other intricacies, such as variation between runs, may be important to a system owner and may not be adequately represented when looking at the final AUC measurement alone. Additionally, the graphs above focus on vehicle speed and not holistic vehicle performance. The attack performed in this research, even after defenses are in place, could have significant impacts on the vehicles that are not accounted for in this analysis. For example, when any accelerator pedal attack message gets past the defense, that message causes the vehicle's throttle body actuator to change positions from what the driver is requesting. This leads to rapid fluctuations in the throttle body actuation signal, which would cause accelerated wear in the best case and could damage the component in the worst case. However, our intention is not to solve issues already facing the implementation of a network layer CAN bus defense. In Table 5, we show the baseline performance of each ML algorithm trained and tested with or without observation dropping (described in Section 3.4). Labels \"drop/drop\" indicate that dropping was used during training and evaluation, and LSTM rows only indicate the test scenario as it must be trained solely on true messages. For the RL model and CNN binary classifier, dropping blocked messages yielded higher mean accuracy than preserving them in future observations, both with and without injected messages. The LSTM model accuracy benefitted from observation dropping only with no injections. The technique also reduced the models' variance, except for the RL model with injections and the LSTM model across both. Our RL approach with observation dropping produced the highest mean accuracy with or without injections and the lowest variance without injections.