# An Energy Management Strategy of Power-Split Hybrid Electric Vehicles Using Reinforcement Learning

Authors: Juanying Zhou; Jianyou Zhao; Lufeng Wang
Journal: Mobile Information Systems
 (2022)

---

## Abstract

With the rapid development of science and technology, the automobile industry is also gradually expanding due to which energy security and ecological security are seriously threatened. This paper was aimed at studying the energy organization strategy of power-split hybrid electric vehicles based on a reinforcement learning algorithm. A power-split hybrid electric vehicle (HEV) combines the advantages of both series and parallel hybrid vehicle architectures by using a planetary gear set to split and combine the power generated by electric machines and a combustion engine. This improves the fuel economy to some extent. However, to increase the fuel economy to a greater extent. This study primarily introduces the hybrid electric vehicle’s structure and presents a reinforcement learning-based management of energy approach for hybrid electric vehicles. It constructs the vehicle power model of HEV and Markov probability transfer model, then designs the energy control strategy based on reinforcement learning, and finally compares it with the energy control strategy based on PID (Proportional-Integral-Derivative). Using MATLAB/Simulink, the cycle conditions of NEDC (New European Driving Cycle) and FTP-75 (Federal Test Procedure) are selected to carry out simulation experiments. The energy management technique suggested in this study, which is based on reinforcement learning, may efficiently enhance the usage rate of automotive gasoline. The replication results show that the fuel consumption per 100 km (kilometers) based on reinforcement learning management strategy is 4.6% and 2.7% lower than the PID management strategy under two working conditions. The economy of fuel of the hybrid electric vehicle is also effectively improved.


---

## Body

## 1. Introduction

The development of traditional vehicles is facing severe challenges. The world is conducting in-depth research for the improvement of the fuel economy and emission. It has attracted the attention of relevant experts too. Hybrid electric vehicle takes the traditional vehicle as the foundation and adds electric system to change the operation condition of the traditional internal combustion engine. It enhances the vehicle’s fuel economy and emission performance. Because of the additional degree(s) of freedom provided by battery energy storage and one or more electric machines, it improves fuel economy by allowing a smaller combustion engine to operate in a higher efficiency region (s). In addition, the battery storage allows for the capture of the energy of braking, which is often squandered as heat in traditional automobiles. At present, hybrid technology is an efficient way to save fuel and is the main research hotspot. The coordinated operation between the two power sources can make the vehicle engine run around the best working point. The engine and motor work together to deliver higher power when the vehicle’s power requirements are high. When the vehicle’s power requirement is moderate, the engine’s output power charges the battery. When the control compulsory by the automobile is relatively small, the automobile power battery provides electric energy for the motor, and the automobile motor drives the automobile. When the car slows down or stops automatically, the car motor can act as a generator to recover the kinetic energy generated by the inertial car. Hybrid electric vehicles do not need to build additional charging stations. In case of machine design, several automakers provide power-split or parallel-series hybrid electric powertrains that enable both series and parallel capability. The engine and generator are connected to the planet carrier and sun gear of a planetary gear set or speed coupler in this arrangement, respectively. The output of the speed coupler is coupled with the output of a second motor/generator through a torque coupler to power the vehicle driveline. Because of the numerous possible power flows in the power-split powertrain, the engine operating point may be regarded separated from the vehicle operating point. In addition, the battery storage system gives you more options for storing and delivering energy. Hence, the HEV has two degrees of freedom, which, when combined with the various modes of operation, allows the vehicle to function more efficiently and, as a result, reduces fuel consumption and emissions. When written in this way, the energy management problem, an optimal control structure becomes a dynamic, nonlinear, and constrained optimization due to the nonlinearities in the dynamic model of the problem the powertrain, as well as a number of equality and inequality restrictions on the control inputs and on the states. Researchers in the past have employed numerical solutions, such as those based on dynamic programming (DP) or have condensed the dynamic optimization issue to an analogous instantaneous optimization, referred to as ECMS that stands for equivalent consumption minimization technique. In this way, the vehicles consume less fuel and give more output power. The innovations of this paper are as follows:(i)
Various HEV designs are described, and vehicle management of energy based on reinforcement learning is used to keep vehicle fuel consumption under control(ii)
Both traditional vehicle energy management and reinforcement learning-based automobile energy management are thoroughly examined, and simulation studies show that the management technique described in this study may greatly reduce automobile fuel consumptionThis research paper is ordered as follows. The related work is first presented and the basic structure of HEVs is discussed in Section3. The vehicle energy management plan based on reinforcement learning is explained in Section 4, and the analysis and concluding remarks are presented in Sections 5 and 6, respectively.


## 2. Related Work

The hybrid electric vehicle’s administration approach of energy is critical in determining whether the vehicle can accomplish successful energy usage while driving to meet the crucial aim of green energy conservation and pollution reduction. Therefore, a management plan based on a particle swarm optimization algorithm is proposed for power-split power vehicles. In addition, the algorithm mainly adopts a double-layer structure. The inner layer uses the strategy of the investigation mode for fantastic fuel consumption to optimize the vehicle working mode and vehicle working place, while the outer layer adopts particle swarm optimization to optimize the equivalent factors in the middle iteration to effectively control the power of the battery. However, this method has too many iterations. It cannot successfully reduce the energy consumption of vehicles [1]. Hence, this paper focuses on the analysis of rule-based hybrid electric vehicle energy management. It effectively points out that the main contradiction between the practicability of power vehicle energy management and vehicle control effect is a serious issue. The research status and development trend of hybrid mode switching control methods such as coordinated control of motor and clutch are deeply analyzed to make the actual value of engine and point set clutch matrix accurate. It is the main research direction of hybrid electric vehicle mode switching control, taking into account some characteristics such as time-varying parameters of the hybrid system and response delay; however, due to the mode switching of this strategy, vehicle fuel consumption has not been effectively controlled [2]. Given the relative limitations of formulating the management of energy method of the hybrid electric vehicle under fixed cycle conditions, more than 20 typical cycle conditions covering the actual driving of the vehicle are selected from AVDI-SOR software, and the fuel consumption and power battery life of the whole vehicle are taken as the comprehensive optimization objectives. The key parameters of vehicle energy management method under different working conditions are gradually optimized by using the optimization algorithm. The optimization results are constructed into an information database, and the energy management strategy of hybrid electric vehicle based on driving conditions is proposed. However, the power battery temperature and working current of this strategy has decreased. The energy consumption of vehicle comprehensive fuel has not decreased [3]. To effectively advance the energy efficacy of HEV power systems, the management of energy of HEV systems is deeply studied from the perspective of optimal energy efficiency. Aiming at the coupling and time variability of the comprehensive efficiency of the hybrid electric vehicle system, this paper constructs the efficiency evaluation model of the hybrid electric vehicle system and evaluates and dynamically modifies the electric energy efficiency stored in the battery of the HEVs. It also has a goal of optimizing the hybrid electric vehicle’s overall efficiency, which is combined with the particle swarm optimization technique to create the vehicle energy management approach. However, due to the complexity of the process, the vehicle energy consumption remains high [4].


## 3. Basic Structure of Hybrid Electric Vehicle

The hybrid vehicle system is composed of engine, motor, transmission, battery, and other equipment. Figure1 explains the power system of the vehicle.Figure 1
Structure diagram of hybrid power system.


### 3.1. Model of Longitudinal Dynamics

The forward power of HEV is mainly provided by the power source. The resistance of the vehicle includes rolling resistanceFr, slope resistance Fg, air resistance Fw, and acceleration resistance Fa. The model of longitudinal dynamics is expressed as
(1)Fr=fmgcosα,Fg=mgsinα,Fw=0.5CdApav2,Fa=δmdv/dt.In formula (1), f is the coefficient of rolling resistance, m is the mass of hybrid electric vehicle, g is the gravity acceleration of vehicle, α is the slope of path, Cd is the coefficient of wind resistance, A is the main area facing the wind, pa is the density of air, v is the vehicle speed, and δ is expressed as the mass coefficient of vehicle rotation.The hybrid electric vehicle model is established. The equation of longitudinal dynamics is shown. The torque output by the power system is expressed as the torque required by the driver:(2)Tv·ηT+Tb=mgfcosα+mgsinα+0.5CdApav2+δmdvdtrw.In formula (2), Tv represents the torque required by the hybrid electric vehicle, ηT represents the efficiency of the total transmission, Tb represents the braking torque, and rw is the rolling radius of the vehicle wheel.


### 3.2. Mechanism Model of Automobile Planetary Gear

The planetary gear set in the automobile model is mostly used to distribute the power of the automobile motor and engine. The coupling reaction occurs between the automobile engine and the planet carrier, the automobile generator and the automobile gear ring, and automobile motor and the automobile sun gear [5, 6]. The formula for the coupling reaction of these factors is
(3)Rrnr+Rsns=Rr+Rsnc,nr=jnw=jvrw.In formula (3), nr represents the gear ring of the automobile, ns represents the sun gear, nw represents the wheel of the automobile, nc represents the rotational speed of the planet carrier, Rr represents the radius of the gear ring of the automobile, and Rs represents the radius of the sun gear. j is the final drive ratio.The coupling of running power of split power vehicle can make the engine control point in the vehicle not controlled by the vehicle [7, 8], ignoring the inertia caused by the planetary gear set’s planetary carrier’s rotation. It is supposed that the coupling shaft in the power vehicle is rigid, and the power system of the vehicle can be expressed by dynamics:
(4)Jednedt=Te−FRs+Rr,Jmdnmdt=Tm−Tdj+FRr,Jgdngdt=Tg+FRs.In formula (4), Je is the moment of engine’s inertia of the hybrid electric vehicle, Jm is the moment of motor’s inertia of the steam, Jg is the moment of inertia of the motor generator, ne is the engine speed of the vehicle, nm is the motor speed, ng is the generator speed, F is the interaction force between gears, Te is the output shaft torque of the engine, Tm is the output shaft torque of the motor, Tg represents the torque of the generator output shaft, and Td is the torque at the output end of the drive train.


### 3.3. Hybrid Vehicle Engine Model

The gas ingesting degree of HEV engine is a function of torque and swiftness, expressed as(5)mf·=fTe,ne.


### 3.4. Hybrid Electric Vehicle Motor Model

The motor efficiencyηm of hybrid electric vehicle is a motor speed function nm and torque Tm of hybrid electric vehicle, expressed as
(6)ηm=fnm,Tm.The power of hybrid electric vehicle motor is(7)pm=Tmnm9550ηm,engine,Tmnmηm9550,engine.


### 3.5. Hybrid Electric Vehicle Battery Model

The basic physical model of HEV battery can be simplified into the voltage of series internal resistance open circuit, ignoring the effect of temperature and considering the change of battery charge state. The amount of battery charge is expressed as(8)dQdt=U0−U02−4RintPbat2RintQ0.In formula (8), U0 represents the OCV of HEV battery; Rint is the battery internal resistance of hybrid electric vehicle; Pbat is the charge and discharge power of hybrid electric vehicle battery; Q0 is the maximum charge of hybrid electric vehicle battery.


## 3.1. Model of Longitudinal Dynamics

The forward power of HEV is mainly provided by the power source. The resistance of the vehicle includes rolling resistanceFr, slope resistance Fg, air resistance Fw, and acceleration resistance Fa. The model of longitudinal dynamics is expressed as
(1)Fr=fmgcosα,Fg=mgsinα,Fw=0.5CdApav2,Fa=δmdv/dt.In formula (1), f is the coefficient of rolling resistance, m is the mass of hybrid electric vehicle, g is the gravity acceleration of vehicle, α is the slope of path, Cd is the coefficient of wind resistance, A is the main area facing the wind, pa is the density of air, v is the vehicle speed, and δ is expressed as the mass coefficient of vehicle rotation.The hybrid electric vehicle model is established. The equation of longitudinal dynamics is shown. The torque output by the power system is expressed as the torque required by the driver:(2)Tv·ηT+Tb=mgfcosα+mgsinα+0.5CdApav2+δmdvdtrw.In formula (2), Tv represents the torque required by the hybrid electric vehicle, ηT represents the efficiency of the total transmission, Tb represents the braking torque, and rw is the rolling radius of the vehicle wheel.


## 3.2. Mechanism Model of Automobile Planetary Gear

The planetary gear set in the automobile model is mostly used to distribute the power of the automobile motor and engine. The coupling reaction occurs between the automobile engine and the planet carrier, the automobile generator and the automobile gear ring, and automobile motor and the automobile sun gear [5, 6]. The formula for the coupling reaction of these factors is
(3)Rrnr+Rsns=Rr+Rsnc,nr=jnw=jvrw.In formula (3), nr represents the gear ring of the automobile, ns represents the sun gear, nw represents the wheel of the automobile, nc represents the rotational speed of the planet carrier, Rr represents the radius of the gear ring of the automobile, and Rs represents the radius of the sun gear. j is the final drive ratio.The coupling of running power of split power vehicle can make the engine control point in the vehicle not controlled by the vehicle [7, 8], ignoring the inertia caused by the planetary gear set’s planetary carrier’s rotation. It is supposed that the coupling shaft in the power vehicle is rigid, and the power system of the vehicle can be expressed by dynamics:
(4)Jednedt=Te−FRs+Rr,Jmdnmdt=Tm−Tdj+FRr,Jgdngdt=Tg+FRs.In formula (4), Je is the moment of engine’s inertia of the hybrid electric vehicle, Jm is the moment of motor’s inertia of the steam, Jg is the moment of inertia of the motor generator, ne is the engine speed of the vehicle, nm is the motor speed, ng is the generator speed, F is the interaction force between gears, Te is the output shaft torque of the engine, Tm is the output shaft torque of the motor, Tg represents the torque of the generator output shaft, and Td is the torque at the output end of the drive train.


## 3.3. Hybrid Vehicle Engine Model

The gas ingesting degree of HEV engine is a function of torque and swiftness, expressed as(5)mf·=fTe,ne.


## 3.4. Hybrid Electric Vehicle Motor Model

The motor efficiencyηm of hybrid electric vehicle is a motor speed function nm and torque Tm of hybrid electric vehicle, expressed as
(6)ηm=fnm,Tm.The power of hybrid electric vehicle motor is(7)pm=Tmnm9550ηm,engine,Tmnmηm9550,engine.


## 3.5. Hybrid Electric Vehicle Battery Model

The basic physical model of HEV battery can be simplified into the voltage of series internal resistance open circuit, ignoring the effect of temperature and considering the change of battery charge state. The amount of battery charge is expressed as(8)dQdt=U0−U02−4RintPbat2RintQ0.In formula (8), U0 represents the OCV of HEV battery; Rint is the battery internal resistance of hybrid electric vehicle; Pbat is the charge and discharge power of hybrid electric vehicle battery; Q0 is the maximum charge of hybrid electric vehicle battery.


## 4. Vehicle Energy Management Strategy Based on Reinforcement Learning

This section explains the energy management model of hybrid electric vehicles, Markov model of power-split hybrid electric vehicle, and energy management strategy based on reinforcement learning. This will help construct the vehicle energy management strategy based on reinforcement learning.


### 4.1. Energy Management Model of Hybrid Electric Vehicle

The main purpose of hybrid electric vehicle energy management is to reduce the consumption of vehicle fuel, improve the economy and practicability of hybrid electric vehicle fuel, prevent the vehicle from overcharging or over-discharging, improve the service life of hybrid electric vehicle battery, and maintain the remaining power of vehicle battery in a stable range, The optimization objective is represented(9)J=∫t0tα·mfue1‐ratetdt+β·soct−soct02.The optimization value function of hybrid electric vehicle is made of two parts:(I)
It represents the consumption of fuel of the HEVs.mfue1‐rate represents the instantaneous fuel consumption rate of hybrid electric vehicle(II)
It represents the difference between the remaining power at the current time and the initial time. Also, it represents the performance of the remaining power of hybrid electric vehicle battery.α and β are two constant factors. By adjusting these two parameters, the management plan of energy of hybrid electric vehicle can have a relative balance in fuel economy and battery remaining power maintenance [9, 10]


### 4.2. Markov Model of Power-Split Hybrid Electric Vehicle

In the problem of vehicle energy management, the randomness of working conditions is easy to cause random changes in vehicle demand power. The dimension of state variables after discretization is very high. To effectively improve the speed of simulation, the probability matrix of state transition is used to simplify the model. The next step is to model the vehicle driving situation as a Markov process. It simply has to do with the current situation and activity. It has no relationship with the historical stat [11, 12]. The standard NEDC operating state is used in this paper, and the speed and necessary power under that situation are presented in Figures 2 and 3.Figure 2
Curve of vehicle speed under working condition.Figure 3
Power demand under working conditions.The nearest neighbor algorithm is used for approximation of the torque of the power demand probability of hybrid electric vehicle:(10)ps,s′=ns,s′ns.In the formula,ps,s′ is the transition from hybrid vehicle state s to s′ probability; ns,s′ is the number generated in the case of change from state s to s′ in the cycle of HEVs; ns is the total number generated by state s.


### 4.3. Energy Management Strategy Based on Reinforcement Learning

In the management strategy founded on reinforcement learning energy, the state variables, action variables, and reward signals are designed and expressed as(11)St=Preqt,vt,SOCt,at=Pmt,(12)R=1mfue1‐ratemfue1‐ratet≠0∩SOCmin≤SOC≤SOCmax,1mfue1−ratet+10mfue1‐ratet≠0∩SOCminorSOC≻SOCmax,1SOC+1mfue1‐ratet=0∩SOCmin≤SOC,1Pbatttmfue1‐ratet=0∩SOC≺SOCmin.In the above formula, vehicle speedvt, battery remaining power, and demand power Preqt are selected as state variables. Motor power Pmt of hybrid electric vehicle is selected as action variables, and reward signal is a function of correlation with battery remaining power and instantaneous fuel consumption rate [13–14]. In the interaction between the agent and its surroundings, the energy management strategy of reinforcement learning can be used as the agent, and the hybrid electric vehicle can be used as the environment. The state St and reward signal Rt at time t from the environment is obtained by the agent. Then, it selects appropriate actions to act on the environment, and it will feedback to the agent the state St+1 and reward signal Rt+1 at the next time. It cycles this process until the best control strategy is learned [14, 15].The basic goal of reinforcement learning is to define the overall future expectation of cumulative return of all states, which may be expressed as(13)V∗s=E∑t=0∞γtrt.In the formula,γ∈0,1 is the reduction factor to ensure the convergence of hybrid electric vehicle in the learning process, E represents the expectation of cumulative return, and V∗s, as a value function, effectively satisfies the Behrman equation.
(14)V∗s=rs,a+γ∑s′=S+∞Psa,s′V∗s′.In formula (14), Psa,s′ is the likelihood of performing action, a from hybrid vehicle state s to state s′, and rs,a refers to the instant reward existing in the current state.The management approach of energy of a hybrid electric car is designed using the Q learning algorithm, which is a classic in reinforcement learning. The key idea is to create a Q-table that corresponds to state and action and then apply the iterative Q-value function to calculate the predicted cumulative discount return [16, 17]. The state value function and update rule of a hybrid electric car can be written as follows in Q-learning:
(15)Q∗s,a=rs,a+γ∑s′∈SPsa,s′maxaQ∗s′,a′,(16)Qs,a⟵Qs,a+ηr+γmaxa′Qs′,a′−Qs,a.In the formula,η represents the learning rate. The greater the value of the learning rate, the faster the convergence speed of reinforcement learning. However, if the value of the learning rate is too large, it is easy to cause learning oscillation and fitting problems. According to Bellman’s optimality principle, the optimal control strategy can be obtained:
(17)π∗s=argamaxQs,a.To balance the relationship between utilization and exploration in reinforcement learning, theε−greedy algorithm is used. The algorithm has a small probability, 1−ε randomly selects the corresponding action, and the probability selection of ε leads to the maximum action of the value function. The pseudocode of Q-learning algorithm wake-up update is as follows:Initialize allQs,a=0. After initializing s,a, repeat the following process according to the hybrid electric vehicle state s, select the corresponding action a according to the green strategy, execute act a, obtain the recompense R and the new state S1, and cycle until the end of s [18, 19]. Repeat the following process: select the corresponding action a to execute action a according to the hybrid vehicle state s and the green strategy, obtain the reward R and the new state S1, and cycle until s ends.The detailed limitations of reinforcement learning are shown in Table1.Table 1
Specific parameter settings of reinforcement learning.
ProbabilityVectorDiscount factorEpisodeCycle index0.870.10.81020000


## 4.1. Energy Management Model of Hybrid Electric Vehicle

The main purpose of hybrid electric vehicle energy management is to reduce the consumption of vehicle fuel, improve the economy and practicability of hybrid electric vehicle fuel, prevent the vehicle from overcharging or over-discharging, improve the service life of hybrid electric vehicle battery, and maintain the remaining power of vehicle battery in a stable range, The optimization objective is represented(9)J=∫t0tα·mfue1‐ratetdt+β·soct−soct02.The optimization value function of hybrid electric vehicle is made of two parts:(I)
It represents the consumption of fuel of the HEVs.mfue1‐rate represents the instantaneous fuel consumption rate of hybrid electric vehicle(II)
It represents the difference between the remaining power at the current time and the initial time. Also, it represents the performance of the remaining power of hybrid electric vehicle battery.α and β are two constant factors. By adjusting these two parameters, the management plan of energy of hybrid electric vehicle can have a relative balance in fuel economy and battery remaining power maintenance [9, 10]


## 4.2. Markov Model of Power-Split Hybrid Electric Vehicle

In the problem of vehicle energy management, the randomness of working conditions is easy to cause random changes in vehicle demand power. The dimension of state variables after discretization is very high. To effectively improve the speed of simulation, the probability matrix of state transition is used to simplify the model. The next step is to model the vehicle driving situation as a Markov process. It simply has to do with the current situation and activity. It has no relationship with the historical stat [11, 12]. The standard NEDC operating state is used in this paper, and the speed and necessary power under that situation are presented in Figures 2 and 3.Figure 2
Curve of vehicle speed under working condition.Figure 3
Power demand under working conditions.The nearest neighbor algorithm is used for approximation of the torque of the power demand probability of hybrid electric vehicle:(10)ps,s′=ns,s′ns.In the formula,ps,s′ is the transition from hybrid vehicle state s to s′ probability; ns,s′ is the number generated in the case of change from state s to s′ in the cycle of HEVs; ns is the total number generated by state s.


## 4.3. Energy Management Strategy Based on Reinforcement Learning

In the management strategy founded on reinforcement learning energy, the state variables, action variables, and reward signals are designed and expressed as(11)St=Preqt,vt,SOCt,at=Pmt,(12)R=1mfue1‐ratemfue1‐ratet≠0∩SOCmin≤SOC≤SOCmax,1mfue1−ratet+10mfue1‐ratet≠0∩SOCminorSOC≻SOCmax,1SOC+1mfue1‐ratet=0∩SOCmin≤SOC,1Pbatttmfue1‐ratet=0∩SOC≺SOCmin.In the above formula, vehicle speedvt, battery remaining power, and demand power Preqt are selected as state variables. Motor power Pmt of hybrid electric vehicle is selected as action variables, and reward signal is a function of correlation with battery remaining power and instantaneous fuel consumption rate [13–14]. In the interaction between the agent and its surroundings, the energy management strategy of reinforcement learning can be used as the agent, and the hybrid electric vehicle can be used as the environment. The state St and reward signal Rt at time t from the environment is obtained by the agent. Then, it selects appropriate actions to act on the environment, and it will feedback to the agent the state St+1 and reward signal Rt+1 at the next time. It cycles this process until the best control strategy is learned [14, 15].The basic goal of reinforcement learning is to define the overall future expectation of cumulative return of all states, which may be expressed as(13)V∗s=E∑t=0∞γtrt.In the formula,γ∈0,1 is the reduction factor to ensure the convergence of hybrid electric vehicle in the learning process, E represents the expectation of cumulative return, and V∗s, as a value function, effectively satisfies the Behrman equation.
(14)V∗s=rs,a+γ∑s′=S+∞Psa,s′V∗s′.In formula (14), Psa,s′ is the likelihood of performing action, a from hybrid vehicle state s to state s′, and rs,a refers to the instant reward existing in the current state.The management approach of energy of a hybrid electric car is designed using the Q learning algorithm, which is a classic in reinforcement learning. The key idea is to create a Q-table that corresponds to state and action and then apply the iterative Q-value function to calculate the predicted cumulative discount return [16, 17]. The state value function and update rule of a hybrid electric car can be written as follows in Q-learning:
(15)Q∗s,a=rs,a+γ∑s′∈SPsa,s′maxaQ∗s′,a′,(16)Qs,a⟵Qs,a+ηr+γmaxa′Qs′,a′−Qs,a.In the formula,η represents the learning rate. The greater the value of the learning rate, the faster the convergence speed of reinforcement learning. However, if the value of the learning rate is too large, it is easy to cause learning oscillation and fitting problems. According to Bellman’s optimality principle, the optimal control strategy can be obtained:
(17)π∗s=argamaxQs,a.To balance the relationship between utilization and exploration in reinforcement learning, theε−greedy algorithm is used. The algorithm has a small probability, 1−ε randomly selects the corresponding action, and the probability selection of ε leads to the maximum action of the value function. The pseudocode of Q-learning algorithm wake-up update is as follows:Initialize allQs,a=0. After initializing s,a, repeat the following process according to the hybrid electric vehicle state s, select the corresponding action a according to the green strategy, execute act a, obtain the recompense R and the new state S1, and cycle until the end of s [18, 19]. Repeat the following process: select the corresponding action a to execute action a according to the hybrid vehicle state s and the green strategy, obtain the reward R and the new state S1, and cycle until s ends.The detailed limitations of reinforcement learning are shown in Table1.Table 1
Specific parameter settings of reinforcement learning.
ProbabilityVectorDiscount factorEpisodeCycle index0.870.10.81020000


## 5. Analysis of Experimental Results

The simulation experiment is carried out on a 24-core workstation. Firstly, the power-split vehicle simulation model is constructed; then, the reinforcement learning code is written. The simulation experiment of hybrid electric vehicle energy management is approved out by using MATLAB/Simulink.The specific working condition of the training algorithm simulated in this paper is the Worldwide Harmonized Light Vehicles (WLTC) cycle. It mainly simulates the city and loop. Rural roads, highways, and other different working conditions will lead to great differences in the speed of cars, but it is more in line with the daily use of car users, as shown in Figure4. The specific working condition of the test algorithm in this paper is the NEDC cycle working condition. It is mainly composed of the cycles of operation in urban areas and suburban areas. It has the phenomena of frequent acceleration and parking and can also accelerate continuously, as shown in Figure 2.Figure 4
Hybrid electric vehicle in WLTC cycle.Figure5 displays the curve of vehicle energy management learning founded on reinforcement learning. It can be observed from Figure 5 that the cumulative reward curve of the reinforcement learning algorithm fluctuates greatly at the initial stage of learning, but with the continuous increase of training rounds, the cumulative reward also increases and the fluctuation decreases gradually. After the 50th round, the cumulative reward begins to converge. This is because the exploration factor of the reinforcement learning algorithm effectively explores the action and then reasonably adjusts the network based on relevant parameters. Figure 6 shows that the exploration factor gradually tends to zero with the increase of training rounds. Because the reinforcement learning algorithm can successfully explore the data in the data memory through high probability actions to prevent the agent from using the data information in the memory, resulting in poor learning impact, a greater vehicle exploration factor must be determined at the outset. Thus, the speed of early convergence of reinforcement learning algorithms is affected. With the increasing number of training rounds, the exploration factor also gradually produces and decreases until it is close to 0, which means that the actions performed by the agent are given by the reinforcement learning controller.Figure 5
Energy management curve based on reinforcement learning.Figure 6
Learning curve of optimization strategy based on reinforcement learning.Figure7 shows the change curve of SOC during the cycle condition test of NEDC based on reinforcement learning energy management control strategy and PID energy management control strategy. It can be observed from the figure that the energy control strategy based on reinforcement learning can better maintain the state of charge (SOC). There is little change in the SOC from the beginning of the working condition to the end of the working condition, while the SOC of the energy control strategy based on PID is relatively poor. The SOC increases significantly from the beginning of the working condition to the end of the working condition, and the change of the overall working condition is more intense than that of reinforcement learning. At about 200 s, the hybrid electric vehicle decelerates. Reinforcement learning mostly employs the excess energy collected from the electric generator to have the vehicle engine operating in the high efficiency region. The PID algorithm, on the other hand, employs the motor to drive the vehicle. Therefore, the SOC of reinforcement learning increases and the SOC of the PID algorithm decreases. At 800 s, reinforcement learning takes priority over the PID algorithm to store electric energy through the motor. Because PID is widely used to drive cars in the early stage, the power output of the engine is used to supplement electric energy during this period, resulting in large fluctuations in SOC and increased energy consumption.Figure 7
Under two control strategies of SOC cycle.To better highlight the differences between the two energy control strategies, Figure8 shows the differences between the two energy control strategies. Figure 8 is the derivation diagram of energy control automobile engine and automobile motor torque based on PID. Figure 9 is the derivation diagram of energy control automobile engine and automobile motor torque based on reinforcement learning. The torque distribution caused by the two energy control methods is quite different. At 850 s, the engine torque of the PID energy control plan is higher than that of the reinforcement learning energy control strategy. It is also very consistent with the curve rise of automobile SOC in Figure 7. It can be clearly seen through the comparison between Figures 8 and 9. Intensive learning automobile energy management uses automobile motor charging to confirm that the automobile engine is in high-efficiency operation at the stage of frequent start and stop before the 800 s. It can successfully regulate the fuel consumption of the automobile and gradually reduce it to improve the economic practicability of the automobile. However, the automobile motor of PID energy control strategy cannot effectively carry out real-time control. In most cases, it is in the state that the automobile motor and the automobile engine work separately.Figure 8
Energy control strategy based on PID for engine and motor torque.Figure 9
Energy control strategy based on reinforcement learning engine and motor torque.After the calculation is converted to equivalent the consumption of fuel, the fuel consumption per 100 km of the two hybrid electric vehicle energy control measurement strategies in the NEDC cycle is displayed in Table2. The 100 km consumption of fuel of the vehicle based on the reinforcement learning energy control strategy is 7.62 l, while the 100 km consumption of fuel of the vehicle based on the PID energy control strategy is 8.08 l. The fuel economy of the reinforcement learning control strategy has been significantly improved compared with the PID energy control strategy, indicating the efficacy of the reinforcement learning energy control strategy. To prevent the contingency of simulation results, Table 3 shows the fuel consumption per 100 km of hybrid electric vehicles under the cycle conditions of ftp75 under both working conditions. The energy control approach grounded on reinforcement learning has also improved the fuel economy of vehicles compared with that based on the PID energy control strategy.Table 2
Fuel consumption per 100 km under NEDC conditions of the two control strategies.
Control strategyFuel consumption per 100 km (L/100 km)Relative fuel economy (%)PID8.08—Reinforcement learning7.624.60Table 3
Fuel consumption per 100 km under FTP75 condition of the two control strategies.
Control strategyFuel consumption per 100 km (L/100 km)Relative fuel economy (%)PID7.67—Reinforcement learning7.452.70It is normal that the performance of vehicle management plan based on reinforcement learning is different in two different working conditions. The data information of training conditions does not completely include all driving conditions, so we can only use the learned control strategy to effectively manage the energy of hybrid electric vehicles. Various test situations have significant variances, resulting in different control effects of the algorithm on nontesting conditions.From the outcomes of the simulation experiment, it can be observed that the energy management of reinforcement learning has very strong practicability and can have a good control effect on the cycle condition. Compared with the vehicle energy management of PID, it can save the participation time, and with the gradual increase of training conditions, the energy management of reinforcement learning can achieve better results.


## 6. Conclusions

To summarize, two MPC-based techniques were used to resolve the fuel reduction problem of power-split hybrid electric vehicles. For the HEV optimum regulator issue, a quadratic cost function is established, and a linear time-varying MPC is used to solve the issue online in the first way. A second charge purpose is included to increase the fuel efficiency by separating the fuel consumption cost into a stage cost and an approximation of cost-to-go as a function of the battery’s state of charge. Based on the concept discussed in this research paper, two important results can be deduced from it. First, the torque of the vehicle is reasonably distributed based on reinforcement learning. Under the condition of theoretical analysis, the energy management of the power vehicle is tested, and two cycle conditions are selected. The test results are obviously better than other energy control strategies. It can effectively reduce the fuel consumption of the vehicle. Second, the research on the management plan of energy of hybrid electric vehicle has reference value. In the follow-up, it will be combined with traffic information, speed prediction and high-precision map, and other information data with reinforcement learning to continuously optimize the energy management strategy of HEV and use some hardware in real vehicle testing to effectively verify the reinforcement learning algorithm proposed in this paper.



---

*Source: 9731828-2022-04-21.xml*