# Energy Management for a Power-Split Plug-In Hybrid Electric Vehicle Based on Reinforcement Learning
## 4 Simulation and Result  
In this article, simulations are conducted based on the Autonomie and Matlab/Simulink. New European Driving Cycle (NEDC), Highway Fuel Economy Test (HWFET) and Urban Dynamometer Driving Schedule (UDDS), shown in Figure 9, are employed to verify the proposed strategy. The selected driving cycles can represent most of the driving pattern under different driving conditions. To compare the performance of the proposed method, the charge depletion/charge sustaining (CD/CS) algorithm is introduced as a benchmark, which is widely employed in actual applications. In addition, the ECMS is also employed to compare the performance of the proposed algorithm. For the CD/DS algorithm, the power distribution of the vehicle can be easily achieved by setting a series of control parameters without any pre-known information of driving conditions. During the CD stage, except for some specific situation, the engine generally remains shut down, and the tractive power is mainly provided by the battery until the SOC drops to a specified lower threshold (e.g., 30%). Then, the vehicle is powered by both the engine and the battery to remain SOC near the specified value under the CS stage. The detailed CD/CS control scheme can be described [12] as:  
Figure 8: Mean discrepancy of the Q-values. Figure 9: Profile of simulation driving cycles. \[P_{bat}=\left\{ \begin{array}{ll}P_{req}&SOC>36\%\\ \min(27804.9,P_{req})&33\%\leq SOC\leq 36\%\\ \min(27804.9\cdot(SOC-0.3)/0.03,P_{req})&30\%\leq SOC\leq 33\%\\ \max(-28157.5\cdot(SOC-0.3)/0.03,P_{req})&P_{req}<0,27\%\leq SOC\leq 30\%\\ \max(-28157.5\cdot(SOC-0.3)/0.03,P_{req}-P_{eng\_max})&P_{req}>0,27\%\leq SOC \leq 30\%\\ \max(-28157.5,P_{req})&P_{req}<0,SOC<27\%\\ \max(-28157.5,P_{req}-P_{eng\_max})&P_{req}>0,SOC<27\%\end{array} \right. \tag{16}\]  
where \(P_{eng\_max}\) represents the maximum power of engine. The ECMS algorithm, as a classical real-time optimization algorithm, transfers the electric consumption of the battery to the equivalent fuel consumption and then tries to minimize the fuel consumption. During each time constant, the vehicle power request is distributed to the battery and the engine according to the minimum principle. By this way, the whole fuel consumption can be reduced and the fuel economy can be improved simultaneously. A typical solution of the ECMS can be formulated based on the Hamilton function, as:  
\[H(x(t),u(t),\lambda(t),t)=F_{rate\_eng}(u(t),t)+\lambda\cdot f(x(t),u(t),t) \tag{17}\]  
where \(\lambda\) is an equivalent factor that can be adjusted dynamically or can be fixed as a constant value. \(x(t)\) and \(u(t)\) are state variables and control variables, respectively. In this paper, \(x(t)\) includes the battery SOC, the vehicle power demand, and the vehicle speed. Similar to before, \(u(t)\) is the battery power. By solving (17), the optimal solution can be found and the final fuel consumption can be obtained. In simulation validation, three standard cycles are selected to splice multifarious and verifiable conditions. Cycle 1 is consisted of two NEDC cycles, one UDDS cycle and two HWFET cycles, Cycle 2 is comprised of two UDDS cycles, two NEDC cycles and two HWFET cycles, and Cycles 3 and 4 includes five and six HWFET cycles. Cycles 5 and 6 are consisted of six and seven UDDS cycles, respectively. The fuel consumption results with the SOC correction [33] are listed in Table 3. It can be found that compared with the CD/CS scheme, the RL based control strategy can effectively reduce the fuel consumption by 10.1%, 9.31%, 4.84%, 4.49%, 5.95% and 5.13% under different driving cycles. Compared with the ECMS, the RL algorithm can gain similar fuel consumption savings. Thus, the validity of RL based algorithm can be proved. More intuitively, Figure 10 shows the battery power comparison with respect to the proposed algorithm, the ECMS and the CD/CS scheme. The power range of the battery based on the RL algorithm is from \(-12\) kW to \(12\) kW, while the battery power based on the CD/CS algorithm ranges from \(-30\) kW to \(5\) kW.