# Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
## 1 Introduction  
An energy management strategy (EMS) is one of the key technologies for hybrid electric vehicles (HEVs) due to its decisive effect on the performance of the vehicle [1]. The EMS for HEVs has been a very active research field during the past decades. However, how to design a highly-efficient and adaptive EMS is still a challenging task due to the complex structure of HEVs and the uncertain driving cycle. The existing EMS methods can be generally classified into the following three categories: (1) Rule-based EMS, such as the thermostatic strategy, the load following strategy, and electric assist strategy [2, 3]. These methods rely heavily on the results of extensive experimental trials and human expertise without the a priori knowledge of the driving conditions [4]. Other related control strategies employ heuristic control techniques, with the resultant strategies formalized as fuzzy rules [5, 6]. Though these rule-based strategies are effective and can be easily implemented, their optimalityand flexibility are critically limited by working conditions and, consequently, are not adaptive to different driving cycles. (2) Optimization-based EMS: some optimization methods employed in control strategy are either based on the known driving cycles or predicted future driving conditions, such as dynamic programming (DP) [7; 8; 9], sequential quadratic programming (SQP), genetic algorithms (GA) [10], the Pontryagin minimum principle (PMP) [11], and so on. Usually, these algorithms can manage to determine the optimal power split between the engine and the motor for a particular driving cycle. However, the obtained optimal power-split solutions are only optimal with respect to a specific driving cycle. In general, it is neither optimal nor charge-sustaining for other cycles. Unless future driving conditions can be predicted during real-time operation, there is no way to imply these control laws directly. Moreover, these methods suffer from the \"curse of dimensionality\" problem, which prevents their wide adoption in real-time applications. Model predictive control (MPC) [12] is another type of optimization-based method. The optimal control problem in the finite domain is solved at each sampling instant and control actions are obtained based on online rolling optimization. This method has the advantages of good control effect and strong robustness. (3) Learning-based EMS: some strategies can learn from the historical data or use the previous driving data for online learning or application [13; 14]. Some researchers propose that traffic information and cloud computing in intelligent transportation systems (ITSs) can enhance HEV energy management since vehicles obtain real-time data via intelligent infrastructures or connected vehicles [15; 16]. Regardless of the learning from historical data or predicted data, these EMS methods also need complex control models and professional knowledge from experts. Thus, these EMS methods are not end-to-end control methods. Reinforcement learning-based control methods have also been used for HEV energy management [17; 18]. However, reinforcement learning must be able to learn from a scalar reward signal that is frequently sparse, noisy, and delayed. Additionally, the sequence of highly-correlated states is also a large problem of reinforcement learning, in addition to the data distribution changes, as the algorithm learns new behaviors in reinforcement learning. The learning-based EMS is an emerging and promising method because of its potential ability of self-adaption according to different driving conditions, even if there are still some problems. In our previous research, online learning control strategies based on neural dynamic programming (NDP) [19], fuzzy Q-learning (FQL) [20], were proposed. These strategies do not rely on prior information related to future driving conditions and can self-tune the parameters of the algorithms. A back propagation (BP) neural network was used to estimate the Q-value which, in turn, tuned the parameter of fuzzy controller [20].