# Energy Management for a Power-Split Plug-In Hybrid Electric Vehicle Based on Reinforcement Learning
## 1 Introduction  
In recent years, as the greenhouse effect and air pollution have become increasingly severe, green energy attracts more attention in all walks of life. In automotive industry, exhaust emission from conventional fuel vehicles is an important factor that causes the environmental pollution. Developing new energy vehicles (NEVs) has shown its significance in reducing emission and lessening induced air pollution. Currently, NEVs can be mainly classified into three types, i.e., fuel cell vehicles, battery electric vehicles (BEVs) and hybrid electric vehicles (HEVs), and they are usually equipped with an energy storage system, such as a battery pack or a super-capacitor [1; 2]. For BEVs, it can be powered purely by the battery pack or the super-capacitor. Plug-in hybrid electric vehicles (PHEVs) are considered to combine advantages of both BEVs and HEVs [3]. Compared with HEVs, the prominent advantage of PHEVs is that the battery pack can be recharged by the external charging plug, thereby supplying certain all electric range (AER). Compared with BEVs, the controller of PHEVs can start the engine to sustain the battery when a certain battery state of charge (SOC) threshold is reached and meanwhile supply the extended driving range. Consequently, it is critical to manage the power distribution between the battery and the engine properly in PHEVs. Energy management strategy (EMS) of PHEVs is responsible for power and energy distribution among different energy storage systems, such as gasoline engine and electromotor. Different controltradeoff of energy management target is mentioned in related literatures [4; 5] including fuel economy improvement [6], and tailpipe emission reduction [7]. Rule based and optimization based methods are mostly considered, as discussed by the authors of [8]. Rule based methods are relatively easier to exploit and are widely employed in practice [9; 10]. In [9], a classified rule based EMS is designed, which emphasizes on different operating modes of PHEVs, and simulation results yields satisfied emission reduction. However, these rule based strategies highly depend on design process and engineering experience, thus leading to longer design time [11]. On the contrary, modern real-time and global optimization based algorithms can be applied with provable optimal guarantee. In particular, dynamic programming (DP), adopted by many researchers, is generally treated as an emblematic algorithm among all the optimal methods [12; 13; 14; 15]. In [12], the investigators proposed an intelligent EMS based on DP, by which numerical simulation results manifest the improved fuel economy dramatically. Quadratic programming (QP) is also a mature algorithm to search for the optimal result with affordable operational budgets [16], compared with DP. Pontryagin minimum principle (PMP) [17] and equivalent consumption minimization strategy (ECMS) [18] are also widely adopted in EMS of PHEVs. In addition, model predictive control (MPC) [19], is extensively investigated as a real-time optimization manner applying to EMS of PHEVs. Furthermore, intelligent algorithms such as simulated annealing (SA) optimization [17], neural network (NN) [20], genetic algorithm (GA) [21] are also employed for EMS of PHEVs in recent years. Nowadays, with development of artificial intelligence (AI) technology, reinforcement learning (RL) is becoming more and more popular in various fields including robotic control, intelligent system, and energy management of power grids. In [22], a parallel control architecture based on the RL technology is applied for robotic manipulation, thereby enabling robots to easily adapt to the environment variation. RL is also introduced in the field of energy management of PHEVs in [23; 24; 25; 26; 27; 28; 29; 30]. In [23], the investigators find that the RL based EMS cannot only guarantee the vehicle dynamic performance, but also improve the fuel economy, and as a result, can outperform stochastic dynamic program (SDP) in terms of adaptability and learning ability.