## 2 Reinforcement Learning for Hybrid Electric Vehicles
### From Dynamic Programming to Reinforcement Learning  
Among ML decision-making techniques, RL represents the family of algorithms featured by an agent interacting with an environment to reach a desired goal [16]. A feedback signal named \"reward\" is passed to the agent at each time step of a training episode as an index of the effectiveness of the action taken.  
The agent-environment interaction can be formalized through a Markov decision process (MDP). A classic formulation of an MDP is completely described by a tuple \(<\)\(\hat{S}\), \(\hat{A}\), r, P, \(\gamma>\) where \(\hat{S}\) is the state space (i.e., set of states); \(\hat{A}\) is the action space (i.e., set of control actions available to the agent); r: \(\hat{S}\times\hat{A}\to R\) is the reward function; P: \(\hat{S}\times\hat{S}\times\hat{A}\rightarrow[0,1]\) is the state-action transition probability function; \(\  
for a time step is completed, the system moves to the following time step \(t+1\) and the operations are repeated iteratively until a terminal state is reached:  
\[e_{T}=(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\ldots,s_{T-1},a_{T-1},r_{T-1},\,s_{T}) \tag{2}\]  
where \(e\) represents the trajectory of the agent-environment interactions within an episode while \(T\) represents the terminal time step. The training process can hence be satisfied by completing multiple episodes.  
The MDP formalism requires the states to satisfy the Markovian property: the transition from \(s_{t}\) to \(s_{t+1}\) depends only on \(s_{t}\) and \(a_{t}\) and not on the complete trajectory from \(s_{0}\) to \(s_{t}\)[16]. Therefore, the transition probability function depends only on \((s_{t},a,s_{t+1})\):  
\[P[S_{t+1}=s_{t+1}\mid S_{t}=s_{t},A_{t}=a_{t}]=P\big{[}S_{t+1}=s^{\prime}\mid S _{t}=s_{t},\,\ldots,\,S_{0}=s_{0},\,A_{t}=a_{t}\big{]} \tag{3}\]  
Both DP and RL are iterative solutions of the Bellman optimality equation [16]. The main difference between the DP optimization and RL lies in the knowledge of the transition probability function \(P\). DP assumes the knowledge of \(P\), so a direct solution of the optimal policy can be obtained offline, without interacting with the environment. Instead, no a priori knowledge about the environment is assumed in RL; therefore, the \(P\) function is neglected from the optimization.  
As a final comment from an engineering perspective, the agent cannot always observe the entire state of the system as a significant number of signals might not be available during real-world conditions. Considering the problem of HEVs real-time control, the sensory input provided to the agent could not constitute the complete view of the environment (i.e., vehicle conditions, actual and future traffic, driver behavior, weather, etc.). In these cases, the agent-environment interactions cannot be modeled through Equation (2) and a distinction between state and \"observation\" needs to be introduced. The state \(s_{t}\), which satisfies the Markovian property of Equation (2), collects all the signals that represents the environment, whereas the observation \(o_{t}\) is a subset of \(s_{t}\) constituted by the set of signals which can be passed to the agent [16]. In other words, the observation should be designed to best approximate the Markovian property about the system state. Even though a theory about Partially Observable MDP [30] has been developed in literature which accounts for the violation of the Markov property for \(o_{t}\), the assumption of this paper is that \(o_{t}\) satisfies the Markovian property.