Proximal Policy Optimization Based Intelligent Energy Management for Plug-In Hybrid Electric Bus Considering Battery Thermal Characteristic

<jats:p>As the performances of energy management strategy (EMS) are essential for a plug-in hybrid electric bus (PHEB) to operate in an efficient way. The proximal policy optimization (PPO) based multi-objective EMS considering the battery thermal characteristic is proposed for PHEB, aiming to improve vehicle energy saving performance while ensuring the battery State of Charge (SOC) and temperature within a rational range. Since these three objectives are contradictory to each other, the optimal tradeoff between multiple objectives is realized by intelligently adjusting the weights in the training process. Compared with original PPO-based EMSs without considering battery thermal dynamics, simulation results demonstrate the effectiveness of the proposed strategies in battery thermal management. Results indicate that the proposed strategies can obtain the minimum energy consumption, fastest computing speed, and lowest battery temperature in comparison with other RL-based EMSs. Regarding dynamic programming (DP) as the benchmark, the PPO-based EMSs can achieve similar fuel economy and outstanding computation efficiency. Furthermore, the adaptability and robustness of the proposed methods are confirmed in UDDS, WVUSUB and real driving cycle.</jats:p>

section_content: As the main transportation for human travel, internal combustion engine (ICE) vehicles consume much non-renewable energy and produce mass pollutant gases [1] [2] [3]. Vehicle electrification is one of the effective ways to alleviate the above problems [4]. Due to the impact of battery technology, hybrid electric vehicles (HEVs) play a crucial role in promoting the development of vehicle electrification [5]. At the same time, plug-in hybrid electric buses (PHEBs) have been gradually applied in urban traffic operations due to their lower fuel consumption and longer driving mileage [6]. 

Since HEVs have a complex power system and extra freedom degrees, the energy management problems have attracted researchers' extensive attention [7]. The existing energy management strategies (EMSs) are mainly divided into rule-based, optimizationbased, and learning-based EMSs [8]. The rule-based EMSs are usually developed from engineering practice based on the engine and motor optimal operating interval, including deterministic rule-based [9] and fuzzy rule-based EMSs [10]. These strategies have strong real-time performance but poor robustness, which makes it challenging to achieve the optimal control effect. 

The optimization-based EMSs, including global optimization-based and instantaneous optimization-based EMSs, aim to reduce vehicle fuel consumption by minimizing the cost function. Global optimization strategies mainly include dynamic programming (DP) [11, 12], Pontryagin's minimum principle (PMP) [13, 14] and convex optimization (CV) action space exploration, avoiding the influence of discrete error on optimization results. A new Actor network is also introduced to separate the agent that learns online from the agent that interacts with the environment. This structure greatly accelerates the training speed and enhances computational efficiency. The Minorize-Maximization algorithm is used to ensure that its performance can be improved with each update policy. Thus, PPO algorithms are insensitive to hyper-parameter changes. Besides, since the PPO algorithms can regularize the Q value, they are not highly dependent on the reward function. After comprehensive analysis and consideration, the PPO-Clip algorithm and PPO-Penalty algorithm are adopted in the energy management of HEV in this paper, which have the strengths of stable training, simple parameter adjustment, and strong robustness. 

In addition to the optimization algorithms, the determination of the objective function is also crucial. The existing EMSs of HEVs mainly target to improve fuel economy and maintain battery charge-sustaining [32, 33] and someone has focused on battery degradation, but without adequately considering the power battery thermal dynamics. The charging and discharging capacity, cycle life, and safety of the battery are dramatically affected by its temperature change. When the battery temperature exceeds the optimal operating range, the battery aging is intensified, the cycle life attenuation is accelerated, and there is a risk of battery spontaneous combustion. 

Motivated by the above literature review and discussion, the PPO-Clip and PPO-Penalty-based EMSs considering battery thermal characteristics are proposed for PHEB. The main contributions are summarized below. 

(1) With a comprehensive consideration for performances in terms of energy saving, battery temperature as well as stable tracking for reference State of Charge (SOC), the PPO-based intelligent algorithm is employed to conduct the research on multi-objective energy management for PHEB. 

(2) The trade-off issue among multiple PHEB energy management objectives is highlighted and addressed by intelligently adjusting the weight coefficients in the training process. 

(3) The battery temperature is online estimated according to its heat generation/ dissipation characteristics, which is further introduced into the PPO-based EMS framework to ensure the battery operation with a rational temperature. 

(4) With respect to DP-based EMS and other RL-based EMSs, extensive comparative simulations are conducted to highlight the effectiveness, superiority, adaptability and robustness of the PPO-based EMSs. 

The rest of the paper is organized as follows. Section 2 presents the powertrain model of the PHEB. Section 3 describes the essential content of the RL algorithm and PPO-based EMSs. Section 4 analyzes the relationship between different objectives and illustrates the simulation results. Section 5 concludes the paper and briefly explains future research. The meanings of some abbreviations are summarized in Abbreviations.
section_name: Introduction

section_content: Referring to the vehicle architecture developed by Zhongtong Group, a single-shaft parallel PHEB model is established, as shown in Figure 1. The engine and motor are attached to the same axle and rotate at the same speed. After coupling the torque, the gearbox and final gear drive the vehicle by reducing speed and increasing torque. When it is fully charged, the power battery provides electric energy to the motor. When the battery's remaining power is insufficient, the mechanical energy is converted into electrical energy by the motor to charge the battery. Thus, there are four working modes of the PHEB: pure electric mode, the engine alone working mode, engine and motor hybrid working mode, and brake recovery mode. The physical parameters of powertrain components are listed in Table 1. The dynamic equation of the vehicle can be described as:

where F t is the driving force of the vehicle, M is the vehicle mass, g is the gravitational acceleration, f is the rolling resistance coefficient, α is the road slope, C d is the air resistance coefficient, ρ is the air density, A is the frontal area of the vehicle, v is the vehicle velocity, δ is the correction factor.
section_name: System Modeling of PHEB

section_content: The fuel consumption of the engine at a specific operating point is related to its speed and torque. By utilizing the varying continuity of the gearbox, the operating point of the engine can be adjusted to the optimal economic zone. The engine map is obtained from bench experiments, as shown in Figure 2. The fuel consumption of the engine per second can be calculated by: where b e is the fuel consumption rate per unit time, T e is the engine torque (Nm), n e is the engine speed (rad/s).
section_name: Engine Model

section_content: The motor used in the PHEB model is a permanent magnet synchronous motor with electric mode, generating mode and idling mode. In electric mode, the battery outputs energy to the motor. In power mode, the motor converts mechanical energy into electrical energy and stores it in the battery. In idling mode, the motor and the battery do not exchange energy. The motor operating efficiency is shown in Figure 3. The output power of the motor can be computed as:

where η m is the motor operating efficiency, T m is the motor torque, n m is the motor speed.
section_name: Motor Model

section_content: The research focuses on lithium-ion batteries which are widely used in electric vehicles in this paper. It is found that the battery's internal resistance is greatly affected by temperature, while the open-circuit voltage varies significantly with different SOC [34]. Consequently, considering battery temperature and SOC changes, a battery internal resistance model is established. For battery packs, the total voltage is as high as 560 V, and the nominal capacity is 90 Ah. The power balance of the battery system is described by:

where P bat is the total battery power consumption, P b is the power flowing into or out of the battery, P l is the power loss due to internal resistance, I bat is the charge and discharge current, R b is the internal resistance. 

The dynamic characteristic of SOC is calculated from the expression:

where
section_name: Battery Electrical Model

section_content: There are two main reasons for the battery temperature rise. The first one is the ohmic resistance heat and the other is the heat generated by a chemical reaction inside the battery. The ohmic resistance heat is irreversible and the chemical reaction heat is reversible. The battery heating rate is given by:

where U t is the terminal voltage, T bat is the battery temperature. 

The battery thermal model is established according to the energy conservation law. The heat balance process can be expressed as:

where m b is the battery mass, c b is the average specific heat capacity, h is the heat exchange coefficient, A b is the heat exchange area, T en is the environment temperature, Q h is the battery heating rate. After equivalent changes, the battery temperature estimation can be obtained by:

where T bat0 is the initial battery temperature, T bat_pre is the battery temperature at the previous moment.
section_name: Battery Thermal Model

section_name: EMSs Based on PPO-Clip and PPO-Penalty

section_content: RL algorithm can solve sequential decision optimization problems, including two main components: agent and environment. The agent can complete specific tasks by learning policy when interacting with the environment. The state of the environment has Markov property and the future state of the system is only related to the current state and has nothing to do with the historical state. Therefore, RL is a Markov decision process (MDP), as a tuple (s, a, P, r). In the tuple, s is the state set, a is the action set, P is the state transition probability matrix and r is the reward function. 

RL agent uses learning as a trial evaluation process. In the beginning, the agent randomly takes an action that impacts the environment. After that, the state of the environment will change and a reward will be generated and fed back to the agent. The agent selects the next action according to the reward and the current state. The interaction process is shown in Figure 5. The principle of action selection is to increase the probability of receiving a larger reward in the future. When the reward tends to a stable maximum value, the agent's task is completed and the optimal results are obtained. In this paper, the agent is the energy and battery temperature management controller. The environment is the PHEB operating condition and powertrain system.
section_name: RL Algorithm

section_content: The agent starts from a specific state until the end of the task, which is called a complete episode. In an episode with T moments, the agent constantly interacts with the environment, forming the following sequence τ:

Since the action taken by the agent may be different at the same state, the sequence τ is uncertain. When the policy function is π θ, the probability of a sequence τ occurrence is:

The return for the sequence τ is the sum of the reward at each moment called R(τ). Therefore, the expected reward can be obtained as:

The policy gradient method is utilized to find the optimal policy. The gradient solution process is:

To reduce the variance of the policy gradient, the advantage function is used to replace the return function. The advantage function is calculated by:

where V θ (s t ) is the state-value function, r k is the reward function in times of k, Q θ (s t, a t ) is the action-value function, γ l is the discount factor. Besides, the idea of importance sampling is adopted and another new policy function π θ is introduced. In this way, the data sampled by interacting with the environment can be reused and the training speed can be improved. The solution process can be expressed as:

where

p θ (a t |s t ) is the importance weight. Since that the parameter distributions of the two policy functions π θ and π θ are close, p θ (s t ) and p θ (s t ) are considered equal. As a result, the update function of the network parameter can be written as:

The core idea of the PPO algorithms is to limit the policy update range by controlling the importance weight. OpenAI and DeepMind use the Clip function and KL penalty to realize, respectively, [35, 36]. 

Method 1 (PPO-Clip): The importance weight is trimmed to a certain extent by introducing the Clip function. The update function becomes:

Method 2 (PPO-Penalty): KL divergence is used to calculate the similarity degree of the action probability distribution. The update function becomes:

The penalty for the difference between θ and θ distribution will be dynamically changed. If the KL divergence value is too large, the penalty will increase. On the contrary, the penalty will reduce if it is small to a certain value.
section_name: PPO-Clip and PPO-Penalty Algorithms

section_content: In this section, the PPO-Clip-based EMS and PPO-Penalty-based EMS will be illustrated in detail. First, define the critical elements of PPO algorithms: state s, action a, and reward function r. 

State s: Considering that battery temperature changes have an impact on driving safety, in addition to battery SOC and vehicle dynamics, battery temperature is also taken as the input state in this paper. 

where acc is the vehicle acceleration. Action a: Thanks to the flexibility of the neural network, the output of continuous action is realized. To reduce the calculation burden, this paper only defines one action, namely the engine torque

Then the motor torque T m is obtained by:

where T req is the demand torque, r wheel is the wheel radius, i g is the transmission ratio of gearbox, i o is the transmission ratio of final gear, and η represents the driveline efficiency. Reward function r: The PPO-Clip-based EMS and PPO-Penalty-based EMS not only target to minimize fuel consumption and keep SOC, but also slow down battery temperature rise. By restraining the battery temperature, the battery life will be prolonged and the operating cost will be reduced [34]. When the agent is in the learning stage, it keeps exploring and learning actions with more enormous rewards [37]. Therefore, the reward function can be described as:

where α, β and γ are the weight coefficients, which will affect the training speed and further affect whether the optimal energy distribution results can be achieved. Moreover, when solving multi-objective optimization problems, the weight coefficients need to be reasonably adjusted until the optimal tradeoff between multiple objectives is realized. 

According to the variance of battery charge-discharge internal resistance characteristic, the SOC must locate in the optimal range to realize high efficiency [28]. In this article, the initial SOC and terminal SOC are set as 0. 8 and 0. 3. 

Lithium-ion batteries typically operate at 0-40 • C. Based on this, the reward function sets the maximum temperature limit T high as 313. 15 K (40 • C). When the current temperature is higher than the temperature upper limit, a negative reward will be punished. 

PPO-Clip and PPO-Penalty algorithms are based on Actor-Critic architecture, including the Actor network and Critic network. The architecture is an online learning algorithm and parameter updates are very slow. To improve the training efficiency, another Actor network is introduced to separate the agent training online from the agent interacting with the environment. Thus, the PPO algorithms are composed of the Actor network parameterized by θ, another Actor network parameterized by θ and the Critic network. Figure 6 shows how PPO-Clip and PPO-Penalty algorithms are implemented in the energy management of PHEB. At first, the Actor network parameterized by θ interacts with the environment to obtain vehicle velocity, acceleration, battery temperature, and SOC. After the interaction is completed, the Actor network parameterized by θ makes decisions based on the current states and calculates the reward of the decision to obtain a series of trajectories τ = {s 1, a 1, s 2, a 2, • • •, s T, a T }. The trajectories are sent to the Actor network parameterized by θ and the Critic network for network parameter update and policy optimization. When the update of the Actor network parameterized by θ reaches a certain number of times, its parameters will be transferred to update the Actor network parameterized by θ. After several iterations, the optimal engine torque distributions are obtained when the expected reward converges to the maximum value [38]. The pseudo-codes of PPO-Clip and PPO-Penalty algorithms are listed as Algorithm 1. 

A fully connected neural network (Deep Neural Network) with one hidden layer is adopted for the Actor-Critic architecture in this paper. For the Actor network, the Relu function is used as the activation function, while the Sigmoid function is used as the output layer to constrain the output action between [0, 1]. The Critic network is similar to the Actor network, except the output layer is the Tanh function that maps state and action to estimated Q values [39]. The parameters of the neural network are described in Table 2.
section_name: Design of Network and Algorithm

section_content: In this section, the performance of the PPO-Clip-based EMS and PPO-Penalty-based EMS will be evaluated in Urban Dynamometer Driving Schedule (UDDS), West Virginia Suburban Driving Schedule (WVUSUB) and real driving cycle collected in Jinan. The UDDS driving cycle is depicted in Figure 7, where the driving time, the average and maximum velocity are 1370 s, 8. 7012 m/s, and 25. 2 m/s, respectively. To realize a complete comparison, the electric consumption of PHEB will be converted into fuel consumption. The equivalent fuel consumption F equ is calculated by:

where E m is the electric consumption, η e is the engine operating efficiency, Q hv is the heating value, ρ fuel is the diesel density. To simulate the complete battery SOC downward trend, the UDDS driving cycle is duplicated four times.
section_name: Simulation Results and Analysis

section_content: Since optimal solutions are diverse, multi-objective problems are difficult to solve. The optimal solution for one objective may be mutually exclusive from other objectives. To balance the multiple optimal results, the weight of each objective needs to be adjusted reasonably. For the energy management problem of PHEBs, it is hoped that the vehicle fuel consumption can be minimized and the power battery can be in the best operating state. To find the appropriate weight coefficient between the three objectives of minimizing fuel consumption, maintaining battery SOC and controlling battery thermal change, an intelligent optimization method is introduced. Firstly, the relationship between each objective is analyzed by adjusting the weight coefficient in proportion. Then, the optimal interval corresponding to different coefficients is determined according to the simulation results. Finally, the reward function is compared to determine the optimal weight coefficient. 

The PPO-Clip-based EMS is taken as an example to adjust the weight coefficients of the three objectives in Equation (22). Firstly, fix α = 1, then set β between 0. 1 × 450 and 1 × 450, and the f soc value is mostly located between 1 and 10. The SOC trajectories with different β are shown in Figure 8. When the β is larger, the constraint on SOC is more strict so that the final SOC will be higher than 0. 3. The simulation results with different β are shown in Table 3. It reveals that fuel consumption increases gradually when SOC rises slowly. Simulation results confirm that the two objectives of improving fuel economy and maintaining SOC are mutually exclusive. By comparison, it can be concluded that the algorithm can achieve better results in fuel economy and SOC maintenance when the β range is between 0. 4 × 450 and 0. 7 × 450. After that, fix α = 1, β = 0. 5 × 450, and let γ vary from 0. 1 to 1. The f bat value is mostly located between 1 and 10. The battery temperature rise curves are shown in Figure 9. It is evident that the larger the weight coefficient γ is, the lower the terminal battery temperature is. When γ = 1, the SOC is the most stable, the temperature rise is the slowest, and the final battery temperature is the lowest. The simulation results between different γ are shown in Table 4. It can be discerned that when SOC decreases, the final battery temperature will increase. It testifies that the two objectives of maintaining SOC and lowering the battery temperature are mutually beneficial. When the γ ranges from 0. 5 to 0. 75, it leads to excellent control results on fuel consumption, battery charge sustaining, and battery temperature rise. After that, set two random numbers and assign values to β and γ randomly. The range of β is (0. 4, 0. 7) and the range of γ is (0. 5, 0. 75). After the same training time, store the current values of β and γ and the reward value of the last training result. After all the training, the reward value is compared, and the value of β and γ corresponding to the minimum reward value is the optimal weight coefficient.
section_name: Tradeoff between Multiple Objectives

section_content: In the RL algorithm, the agent tends to choose the action with an increased reward after a period of exploration and learning. The mean reward distributions are shown in Figure 10. In the beginning, both PPO-Penalty-based EMS and PPO-Clip-based EMS are in the stage of continuous exploration and their mean reward keeps rising. After that, the mean reward gradually tends to be stable. For PPO-Clip-based EMS, the previous exploration is sufficient. After stabilization, the mean reward is higher than that of PPO-Penalty-based EMS. From Episode 30 to Episode 50, PPO-Penalty-based EMS has a small section of invalid learning. In this case, the agent always outputs boundary values. After a period of self-adjustment, the mean reward gradually shows an upward trend and reaches a stable state. Figure 14 shows the engine working points of different strategies. In the DP-based EMS, the engine working points are denser and distributed between the maximum and optimal torque. The PPO-Penalty-based EMS and DDPG-based EMS have a large number of working points, most of which are located in the low fuel consumption area. For DQNbased EMS, the output engine torque is relatively single and centralized, which may be affected by the limited discrete action variable. Compared with the other algorithms, PPO-Clip-based EMS has fewer engine working points, which implies less fuel energy and more electric energy will be utilized. There is no single solution for multi-objective optimization problems to optimize each objective simultaneously. In this case, the objective function is said to be conflicting, and there exists a (possibly infinite) number of Pareto optimal solutions. To solve the above problems, researchers usually set the weights of each goal according to the actual needs. Thus, the compromise of different goals can be achieved through quantification. In this paper, the weight of fuel consumption is set as the maximum to find a set of solutions as close as possible to the Pareto optimal domain. Despite that the PPO-based EMSs achieve a distinct advantage in fuel economy, the final SOC is lower than the other strategies. It is proved that in the multi-objective energy management problems of HEVs, the improvement of fuel economy is at the expense of SOC stability [40]. 

The fuel consumption and the computing time to run 200 episodes (DRL) are listed in Table 6. As shown in Table 6, the PPO-Penalty takes 1435 s for 200 training sessions, which is 200 repeated driving cycles. So running one driving cycle generally takes 1435/200 = 7. 175 s. This is the time standard that can meet the requirements of real-time online control for the EMS of HEVs. In contrast, the 9504 s consumed by the DP algorithm is the time spent on running one driving cycle. It can be seen that although the DP algorithm has optimal control performance, it consumes a lot of computing time and cannot meet the real-time control requirements of vehicles. DP is limited by discrete variables and the optimization results are greatly affected by discrete precision. However, if the discrete precision is improved, the calculation time will increase. On the contrary, due to the flexibility of the neural network, PPO-Clip-based EMS and PPO-Penalty-based EMS can sufficiently explore continuous action space to obtain the optimal energy distribution results. Since PPO-based EMSs separate the agent that trains online from the agent that interacts with the environment, the computing time is greatly reduced. As a result, the trained PPO-based EMSs can be used as vehicle controllers, making real-time online applications possible. In practical applications, the vehicle velocity collected by sensors will inevitably be corrupted by noise. To verify the robustness of the PPO-based EMSs to unknown driving cycles, another driving cycle gathered in Jinan is used to train the EMSs, where the driving time, the average and maximum velocity are 6000 s, 7. 7197 m/s, 21. 6667 m/s, respectively. The corrupted driving cycle is shown in Figure 18
section_name: Effectiveness of EMSs Based on PPO-Clip and PPO-Penalty

section_content: To explore the suitable algorithm applied to the multi-objective energy management optimization problem of PHEB, the PPO-Clip-based EMS and PPO-Penalty-based EMS are investigated in this paper. In addition to maintaining SOC and improving fuel economy, the battery temperature is also taken into account as the optimization objective to keep the battery in optimal working condition. After assigning the proper weights among the three objectives, extensive comparisons are made for further verification. Compared with the original PPO-based EMSs, the proposed EMSs provide effective control of power battery temperature during driving, ensuring that the terminal battery temperature has a lower value. Besides, PPO-based EMSs can realize faster computing speed, lower fuel consumption, and slower battery temperature rise in comparison with DQN-based EMS and DDPG-based EMS. At the same time, the adaptability and robustness of the proposed EMSs are demonstrated under UDDS, WVUSUB and the real driving cycle. It can be concluded that PPO-Clip-based EMS and PPO-Penalty-based EMS express great talent in a comprehensive performance. 

There are two aspects to improve the proposed EMSs in our future work. Firstly, a hardware-in-the-loop experiment and real vehicle validation will be conducted to improve the performance of the PPO-based EMSs. Secondly, real-time energy distribution will be realized online by connecting cloud data to promote practical applications.
section_name: Conclusions

section_content: Funding: This work is supported by the National Natural Science Foundation for Automotive Industry Innovation Joint Fund under Grant U1864205 and Shandong Provincial Key Research and Development Program ( Major Scientific and Technological Innovation Project ) (NO. 2019JZZY020814 ).

section_content: Data Availability Statement: Not applicable.

section_name: Conflicts of Interest:

section_content: The authors declare no conflict of interest.
section_name: Conflicts of Interest:

section_content: The following nomenclature and abbreviations are used in this manuscript:
section_name: Abbreviations