# Energy Management for a Power-Split Plug-In Hybrid Electric Vehicle Based on Reinforcement Learning
## 3 Reinforcement Learning for Energy Management
### Reinforcement Learning Algorithm  
RL, as a significant machine learning method, can conduct repeated explorations in which the agent takes a series of actions in its environment to maximize its designated benefits. The agent-environment interaction for RL is illustrated in Figure 5.  
The agent-environment interaction can be regarded as a Markov decision process, and the RL mainly focuses on solving the Markov decision process based on a series of iteration. In this paper, the state variable \(s\in S\) includes the power request, SOC and the vehicle speed and the action variable \(a\in A\) is the battery power. The reward function \(r\), which evaluates the current action, is defined as the immediate fuel consumption of the engine. The object function could be written as the total reward for the finite future at each state, which can be described as:  
\[V^{*}(s)=E\left(\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right) \tag{11}\]  
Figure 4: The transition probability map. (**a**) The transition probability map at V = 30 km/h; (**b**) The transition probability map at V = 80 km/h. Figure 5: The agent-environment interaction. where \(\gamma\in[0,1]\) is the discount factor to guarantee convergence of the agent during the learning process. Since any state is different and each state is unique, the object function can be reformulated as:  
\[V^{*}(s)=\min_{a\in A}(r(s,a)+\gamma\sum_{s^{\prime}\in S}p_{sa,s^{\prime}}V^{*}( s^{\prime})) \tag{12}\]  
where \(p_{sa,s^{\prime}}\) indicates the transition probability of state variables that change from \(s\) to \(s^{\prime}\) based on action \(a\), and \(r(s,a)\) indicates the reward of applying action \(a\) to transfer from \(s\) to \(s^{\prime}\). The optimal control strategy is determined by [PERSON]'s principle:  
\[\pi^{*}(s)=\operatorname*{argmin}_{a}(r(s,a)+\gamma\sum_{s^{\prime}\in S}p_{sa,s^{\prime}}V^{*}(s^{\prime})) \tag{13}\]  
As a popular candidate of RL algorithms, the QL algorithm is simple and easy to implement [32], and has been widely employed to solve the optimal value function of MDP. The QL algorithm can obtain a strategy to maximize the sum of expected discounted rewards by directly optimizing an iterated value function Q. According to the updated Q value, the agent needs to examine every action in each iteration to make sure that the learning process can converge. In terms of these merits, we employ the QL algorithm as the kernel algorithm to train, learn and finally achieve the energy management of PHEVs. In the QL algorithm, the Q value, i.e., the state-action value, can be written as:  
\[Q^{*}(s,a)=r(s,a)+\gamma\sum_{s^{\prime}\in S}p_{sa,s^{\prime}}\min_{a}Q^{*}(s ^{\prime},a^{\prime}) \tag{14}\]  
Furthermore, the updated rule of Q value can be described as:  
\[Q(s,a)\gets Q(s,a)+\eta(r+\gamma\min_{a^{\prime}}Q(s^{\prime},a^{\prime}) -Q(s,a)) \tag{15}\]  
where \(\eta\in[0,1]\) is a decaying factor. According to the above discussion, the proposed method consists of a simplified vehicle model, a transition probability matrix, a reward matrix and the QL control strategy, where the reward matrix is computed via the simplified vehicle model and the control strategy is calculated according to the power transition matrix, the reward matrix and the QL algorithm feedback. Table 2 lists the pseudocode of the QL algorithm, and it can clearly illustrate the iterative process of QL algorithm. The optimal control strategy is derived through the iterative process shown in Table 2. Figure 6 summarized the detailed procedures of QL in Matlab [19]. First, the QL algorithm and the MDP as well as the related parameters are combined and discretized. Then, the power transition matrix is calculated based on the driver model. Based on the discrete variables and the simplified PHEV model, the reward matrix R is calculated. After iteration, the QL algorithm can be applied successfully to find the optimal energy management solution.