# Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
## 1 Introduction
However, it also requires designing the fuzzy controller, as well as professional knowledge. Deep reinforcement learning (DRL) has shown successful performance in playing Atari [21] and Go games [22] in recent years. The DRL method is a powerful algorithm to solve complex control problems and handle large state spaces by establishing a deep neural network to relate the value estimation and associated state-action pairs. As a result, the DRL algorithm has been quickly applied in robotics [23], building HVAC control [24], ramp metering [25], and other fields. In the automotive field, DRL has been used for lane keeping assist [26], autonomous braking system [27], and autonomous vehicles [28]. However, motion control of autonomous vehicles needs very high precision from our perspective. The mechanism of DRL has not been explained very deeply and may not meet this high requirement. Nevertheless, DRL is a powerful technique that can be used in HEV EMS in this research as it concerns the fuel economy compared to the control precision. A DRL-based EMS has been designed for plug-in hybrid electric vehicles (PHEVs) [29]. This is the first time DRL has been applied to a PHEV EMS. However, there are several problems in this study: (1) The learning process is still offline, which means that the trained deep network can only work well in the same driving cycle, but would not be able to obtain good performance in other driving conditions. As a result, this method can be used in buses with fixed route however it is not acceptable for vehicles with route variation; (2) The immediate reward is important as it affects the performance of DRL. The optimization objective is the vehicle fuel economy, but the reward is a function based on the power supply from the engine. The relationship between fuel economy and engine power is complex and the paper lacks the ability to justify this phenomena; (3) The structure of deep neural network can be well designed by fixing the Q targets network, which can make the algorithm more stable. In this research, an energy management strategy based on deep reinforcement learning is proposed. Our work achieves good performance and high scalability by (1) building the system model of the HEV and formulating the HEV energy management problem; (2) developing a DRL-based control framework and an online learning architecture for a HEV EMS, which is adapted to different driving conditions; and (3) facilitating algorithm training and evaluation in the simulation environment. Figure 1 illustrates our DRL-based algorithm for HEV EMS. The DRL-based EMS can autonomously learn the optimal policy based on data inputs, without any prediction or predefined rules. For training and validation, we use the HEV model built in ADVISOR software (National Renewable Energy Laboratory, Golden, CO, USA). Simulation results reveal that the algorithm is able to improve the fuel economy while meeting other requirements, such as dynamic performance and vehicle drivability. The proposed DRL-based EMS uses a fixed target Q network which can make the algorithm more stable. The immediate reward is a function directly related to fuel consumption. More importantly, a DRL-based online learning architecture is presented. It is a critical factor to apply the DRL algorithm in HEV energy management under different driving conditions. The rest of this paper is organized as follows: Section 2 introduces the system model of HEV and describes the mathematics formulation of HEV EMS. Section 3 explains our deep reinforcement learning-based control strategy, including offline learning and online learning application. The experimental results are given in Section 4, followed by the conclusions in Section 5.