## 3 Optimization-Based Strategies
### Data-Driven Strategies
#### 3.3.2 Reinforcement Learning
Additionally, it can avoid the gradient loss being too fast during the back propagation in the BP neural network. By maximizing the accumulated reward value obtained by the agent from the environment, DQL-based energy management for both minimizing the fuel consumption and keeping the battery SOC stable over a specific time horizon can be developed. A DQL-based energy management strategy for a power split hybrid electric bus (HEB) was proposed in [98]. An experience pool in the form of a quadruplet was defined to store the data needed by the neural network. Furthermore, the optimality and adaptability of the DQL-based strategy on the different driving cycles were verified by comparison with the DP-based strategy. Compared with the Q learning strategy with the same model, the strategy with DQL had a better performance in terms of training difficulty and the influence of different state variables on the action value function. Based on the DQL method and considered as a breakthrough in the field of reinforcement learning, a deep reinforcement learning (DRL) framework for optimizing the fuel economy of HEVs was put forward in [99], which is approximately optimal, model-free, and has no need to have any prior knowledge of the driving cycle. The DRL technique consists of an offline deep neural network and an online deep Q-learning network, and can handle the high dimensional state and action space. To achieve practical application, the required power can be modeled as a stationary Markov model, and the power transition probability matrix can be calculated by the nearest neighborhood method using the RL-based offline strategy. Ref. [100] proposed a RL-based power management control strategy for PHEV, taking optimal fuel economy, battery life, real-time application, and adaptivity in different conditions into account. This strategy can have a good performance in limiting the maximum discharge current and optimizing the system efficiency. To seek the optimal control, different forgetting factors and Kullback-Leibler (KL) [97; 99; 101] divergence rates, which decide whether to update the power management strategy, were discussed. Simultaneously, minimizing the tradeoff between fuel consumption and computational efficiency for HEVs can be achieved with respect to two different aspects. On the one hand, the dimension of action variable can be decreased by setting the engine to operate in a specified area, which contributes to decreasing the training difficulty of the deep neural network. On the other hand, a predictive energy management strategy via a synergy of velocity prediction and RL can be developed. In [102], the nearest neighbor and fuzzy encoding with two transformations (an M-dimensional possibility vector and the proportional possibility-to-probability) was employed to achieve velocity prediction. Furthermore, the transition probability of the vehicle velocity was estimated using the maximum likelihood estimator. The results of an HIL experiment verified that the predictive controller was implementable in real time and had lower fuel consumption than the rule-based strategy.