## 5 Future Work  
Several advancements of this research are required to realize the AICA vision within the CAN bus, let alone an entire vehicle platform. Future work in RL experimentation would include the addition of an inject message action so the agent can handle attacks that block legitimate messages, refinement of the learning process through tuning of RL parameters and better reward functions influenced by QMoCR measurements, addition of online learning for the agent to adapt in real or near-real time, and increased complexity of the training environment to the level of an actual vehicle platform. Additionally, to make such a solution viable for implementation within a vehicle platform, we must change from an on-path (i.e., man-in-the-middle) to an off-path (i.e., man-on-the-side) defense architecture through techniques such as physical layer fingerprinting [35] for message authentication and non-protocol-compliant bus interaction (e.g., CAN-stomping [36]) for blocking, injection, or masquerading defense actions.  
A realistic training environment is essential for a successful defense agent. To provide a suitable training environment without requiring real-time execution, we created a simulation of the PASTA system (see Figure 7). The simulator replicates the CAN bus layout using Linux Virtual CAN (VCAN) interfaces, message sending, and receipt, including message timing, and estimates the physics of the vehicle platform. It also allows the automation of human drivers and associated vehicle drive cycles, as well as software interfaces for network attack creation and injection. We trained a defense model within our simulation environment, but it performed poorly when evaluated in the hardware PASTA environment. We believe this is due to subtle differences between the simulation and hardware environments, such as a drift in message ordering and timing on hardware arising from the non-determinism of multiple processes. Training models, despite this discrepancy, a task known as _sim2 real transfer_[37], is important because any ultimate use of this technology in a real vehicle will require training in simulation to obtain the state space coverage needed for policy optimization. Therefore, additional research is needed into sim2 real transfer techniques for PASTA, such as domain randomization, domain adaptation, or algorithmic techniques, such as perturbing model weights and meta-learning.  
While we evaluated some out-of-distribution learning and have discussed the need for sim2 real transfer techniques, another similar avenue we would like to explore is automatic  
Figure 7: Simulated PASTA training environment.  
curriculum learning via multi-agent interactions. It has been shown that auto-curricula arise when multiple agents are able to learn in a shared, competitive environment [38]. For the CAN bus defense task, one agent may learn to defend (as we do here) while another agent learns to attack. We should expect this system to generate novel attacks if set up appropriately such that the defender agent will generalize better at test time to unknown attacks.  
Finally, it would be worth further exploring the utility of additional KPP evaluations and even additional cyber-resilience measurements to complement and expand upon the information conveyed through an area under the curve measurement.