## 3 Optimization-Based Strategies
### Data-Driven Strategies
#### 3.3.2 Reinforcement Learning  
As a new research hotspot in the field of machine learning and artificial intelligence (AI), reinforcement learning (RL) [97] is the system learning from environment to behavior mapping to maximize the value of the enhanced/reward signal. In recent years, the RL-based energy management strategy has been applied to achieve optimal fuel economy of HEVs in real-time. The schematic diagram of the RL-based strategy is shown in Figure 10.  
Figure 10: Depiction of the schematic diagram of the RL-based strategy. Q learning, as a famous RL algorithm, can achieve satisfactory control performance that does not require an environmental model to strengthen learning. The environment can be regarded as a discrete Markov process with a finite state. The agent can select an action \(a\) within the finite action set at each step, and then the environment accepts the action to transfer the state and give the reward \(r_{t}\). The task of the agent is to determine an optimal strategy to maximize the expected value of the total discount reward signal. The idea of Q learning is to directly optimize the Q function to find a strategy that maximizes the sum of expected discount rewards. The Q function is defined as a discount cumulative enhancement value when the action \(a\) is performed in the state \(s\) under the optimal action sequence, that is,  
\[Q_{t+1}(s_{t},a_{t})=r_{t}+\beta\underset{a_{t}}{\max}\{Q(s_{t+1},a_{t})\} \tag{21}\]  
where \(\beta\) is the discount factor. In energy management based on Q learning, vehicle speed, battery SOC, or engine speed can be selected as the state variables, and the torque-split ratio between engine and motor, engine torque or motor torque can be selected as the control variables. The objective is to minimize the sum of fuel consumption at each moment; therefore, the reciprocal of fuel consumption at each time step can be defined as the immediate reward. The neural network can be used to approximate Q function, and outputs the corresponding action of Q learning. The process of Q learning is as follows:  
1. Initialize the Q network, including selecting the number of neurons in each layer, initializing network connection weights, etc. ;
2. Obtain the state \(s_{t}\) at the moment \(t\) and calculate the Q function of each action \(Q(s_{t},a_{i})\);
3. Select an action \(a_{i}\) and obtain the next state \(s_{t+1}\) and the enhanced signal \(r_{t}\);
4. Calculate \(Q^{\prime}_{t}=r_{t}+\beta\underset{a_{t}}{\max}\{Q(s_{t+1},a_{t})\}\), and then adjust the network weights to minimize the error \(\Delta Q_{i}\) until the error satisfies \(\Delta Q_{i}\leq\delta\), where \(\Delta Q_{i}\) is defined as \(\Delta Q_{j}=Q^{\prime}-Q_{i}\) (\(i=j\)). The weight vectors of the Q network can be regulated based on the gradient-based adaptation algorithm. In [97], a Q learning-based EMS was proposed for PHEVs to optimize the power-split control in real time, which would address the trade-off between real-time performance and optimal energy savings. The simulation of a real-world commute trip shows that about a 12% fuel savings can be achieved without taking charging opportunities into account. However, the Q learning-based strategy would lead to a sharply increasing computational burden and bad convergence ability during the training process due to the \"curse of dimensionality\", which makes the Q-learning strategy difficult in practical application for the energy management of HEVs. To solve the above problem, the deep Q learning (DQL) algorithm with deep neural network (DNN) was developed. Unlike Q learning, a deep neural network (DNN) is employed, and is well trained to approximate the action value function (Q function). Common deep learning networks generally include convolutional neural networks and recurrent neural networks. As for the convolutional neural network, it can handle high-dimensional data well, and weight sharing can reduce the complexity of the network.