## 3 Methods
### Agent Design and Training Using Reinforcement Learning  
RL inherently includes many of the AICA components, as discussed in Section 3.2. In an ideal world, with vast amounts of training data and model refinement, RL should be able to achieve strong policy optimization that combines the effectiveness of various static algorithms with an understanding of the typical behavior of the contextual environment and provides at least some ability to succeed in novel situations. Combined with eventual online learning capabilities, it should be possible to realize the full AICA vision. An RL agent is defined as a mapping \(\pi\colon S\mapsto A\), called a _policy_, from state \(s\) to action \(a\). The policy is defined on an environment represented by a Markov decision process (MDP) \(\langle S,\ A,\ P,\ R\rangle\) with state space \(S\), action space \(A\), transition function \(P(s,\ a,\ s^{\prime})=\Pr(s_{t+1}=s^{\prime}\mid s_{t}=s,\ a_{t}=a)\), and reward function \(R(s,\ s^{\prime})\). The transition function \(P\) describes the environment's probability of transition from state \(s\) to \(s^{\prime}\) when the action \(a\) is chosen. The reward function \(R\) provides an immediate scalar value to the agent when transitioning from state \(s\) to \(s^{\prime}\). The optimization objective for the RL agent is then to maximize the expected _return_, the cumulative discounted reward across the (potentially infinite) time horizon. The expected return is expressed as \(\mathbb{E}\big{[}\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},\ s_{t+1};\ a_{t})\big{]}\), with action \(a_{t}\ =\ \pi(s_{t})\) and discount factor \(\gamma\in(0,1]\). See [30] for more foundational information on RL. In our problem setup, the RL agent observes a fixed-length history of messages as a 2D image (see Figure 3). The image transformation is particularly useful, so we may use a CNN architecture in the agent's value and policy networks, as in [31]. Specifically, each row encodes a single Chassis ECU emanated message with a single pixel representing the one-hot encoded message type, with 15 types in total. The value of the pixel is linearly scaled from 128 to 255 according to the time delta since the last message, computed by \(128\cdot\left(1+\frac{0.001-S}{0.001}\right)\) up to a maximum time \(\delta\) of 0.001 s = 1 ms represented by a value of 128. If no message is received within 1 ms, a blank row is inserted. The decoded message values are represented as linearly scaled values between 0 and 255. Since CAN messages can have multiple data fields, we allowed for representing up to 8 decoded values per message. Following the decoded messages, we encode the data deltas between the current and previous messages of this type with a ceiling of 255. Message history is encoded in the row dimension, and we use a history of 100 messages. The evaluation image--used solely for human understanding of the learning process--includes an agent action column indicating whether the model decided to block or allow, with color coding indicating decision correctness. The action space \(A\) is a set of two actions: allow or block the incoming message. The transition function \(P\) receives the action from the agent, transitions the environment to a new state and gives the agent a new observation. Specifically, if the message is blocked, then it is discarded from the message history going forward. This paradigm is called _observation dropping_ in our results and tends to outperform including these messages in history. Importantly, observation dropping allows the agent to affect its observation space, which converts this classification task into a true sequential decision-making problem for which RL algorithms are suited. If evaluating with hardware in the loop, an allowed message will be emitted to the vehicle bus. The next message will then be received by the environment, and the observation will be updated with the message type, content, and timing information. During training, there is a probability \(p\) of the next message being malicious.