# Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
## 2 Problem Formulation  
The prototype vehicle is a single-axis parallel HEV, the drivetrain structure of which is shown in Figure 2. The drivetrain integrates an engine, an electric traction motor/generator, Ni-Hi batteries, an automatic clutch, and an automatic/manual transmission system. The motor is directly linked between the auto clutch output and the transmission input. This architecture provides the regenerative braking during deceleration and allows an efficient motor assist operation. To provide pure electrical propulsion, the engine can be disconnected from the drivetrain by the automatic clutch. We have  
Figure 1: Deep reinforcement learning (DRL)-based framework for HEV EMS. adopted the vehicle model from our previous work [19; 20] for this research. The key parameters of this vehicle are given in Table 1.  
In the following, key concepts of the DRL-based EMS are formulated:  
**System state:** In the DRL algorithm, control action is directly determined by the system states. In this study, the total required torque (\(T_{dem}\)) and the battery state-of-charge (SOC) are selected to form a two-dimensional state space, i.e., \(s(t)=\left(T_{dem}(t),SOC(t)\right)^{T}\), where \(T_{dem}(t)\) represents the required torque at time \(t\), and \(SOC(t)\) represents the battery state of charge at time \(t\). **Control action:** The decision-making on the torque-split ratio between the internal combustion engine (ICE) and battery is the core problem of the HEV energy management strategy. We choose the output torque from the ICE as the control action in this study, denoted as \(A(t)=T_{e}(t)\), where \(t\) is the time step index. \(T_{e}(t)\) should be discretized in order to apply the DRL-based algorithm, i.e., the entire action space is \(A=\left\{A^{1},A^{2},...,A^{n}\right\}\), where \(n\) is the degree of discretization. In this research, we consider \(n\) as 24. The motor output torque \(T_{m}(t)\) can be obtained by subtracting \(T_{e}(t)\) from \(T_{dem}(t)\). **Immediate Reward:** Immediate reward is important in the DRL algorithm because it directly influences the parameters tuning of the deep neural network (DNN). The DRL agent is always trying to maximize the reward which it can obtain by taking the optimal action at each time step. \begin{table}
\begin{tabular}{c c} \hline \hline
**Part or Vehicle** & **Parameters Value** \\ \hline Spark Ignition & Displacement: 1.0 L \\ (SI) engine & Maximum power: 50 kW/5700 r/min \\  & Maximum torque: 89.5 Nm/5600 r/min \\ \hline Permanent magnet motor & Maximum power: 10 kW \\  & Maximum torque: 46.5 Nm \\ \hline \multirow{3}{*}{Advanced Ni-Hi battery} & Capacity: 6.5 Ah \\  & Nominal cell voltage: 1.2 V \\  & Total cells: 120 \\ \hline \multirow{2}{*}{Automated manual transmission} & 5-speed \\  & GR: 2.2791/2.7606/3.5310/5.6175/11.1066 \\ \hline Vehicle & Curb weight: 1000 kg \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the HEV parameters. Figure 2: Drivetrain structure of the HEV. Therefore, the immediate reward should be defined according to the optimization objective. The control objective of the HEV EMS is to minimize vehicle fuel consumption and emissions along a driving mission. Meanwhile, the vehicle drivability and battery health should be satisfied. In this work, we focus more on fuel economy of the HEV; the emissions are not taken into consideration. Keeping this objective in mind, the reciprocal of the ICE fuel consumption at each time step is defined as the immediate reward. A penalty value is introduced to penalize the situation when the SOC exceeds the threshold. Immediate reward is defined by the following equations:  
\[r^{a}_{ss^{\prime}}= \begin{cases}\frac{1}{C_{ICE}}&C_{ICE}\
eq 0\cap 0.4\leq SOC \leq 0.85\\ \frac{1}{C_{ICE}+C}&C_{ICE}\
eq 0\cap SOC<0.4\text{ or SOC}>0.85\\ \frac{2}{Min_{C_{ICE}}}&C_{ICE}=0\cap 0.4\leq SOC\\ -\frac{1}{C}&C_{ICE}=0\cap SOC<0.4\end{cases}