# Transfer Deep Reinforcement Learning-Based Energy Management Strategy for Plug-In Hybrid Electric Heavy-Duty Trucks under Segmented Usage Scenarios

Authors: Qi Han; Xuan Zhao; Jian Ma; Shu Wang; Kai Zhang; Hua Chai; Yuzhou Yang; Zichen Zheng
Journal: International Journal of Energy Research
 (2023)

---

## Abstract

Energy management strategy (EMS) is a way to reduce the energy consumption of hybrid power systems. This article proposes a unique deep reinforcement learning- (DRL-) based EMS for plug-in hybrid electric heavy-duty trucks (PHETs), combining driving cycle pattern recognition (DPR) and deep transfer learning (DTL). The proposed EMS can cope well with the complex usage scenarios of PHETs and the difficulty of generating EMS. While ensuring the minimum overall driving cost, the strategy can improve the convergence speed of the DRL method and the generalizability under segmented usage scenarios. Firstly, representative driving cycles that reflect different usage scenarios are constructed based on a naturalistic data-driven method. Secondly, a plug-in hybrid electric heavy-duty truck (PHET) driving pattern recognizer based on a learning vector quantization neural network (LVQ) is built. Thirdly, the deep deterministic policy gradient (DDPG) algorithm is innovatively combined with the DTL algorithm. The pretrained neural network in the corresponding usage scenarios is transferred to the natural driving cycles based on DTL. Moreover, the proposed EMS gives an emphasized consciousness on the battery degradation cost. Finally, the strategy is tested under natural driving cycles in different usage scenarios and proven through comparison with the current state-of-the-art techniques, deep reinforcement learning-based strategy, and dynamic programming (DP). The results show that the proposed strategy outperforms existing cutting-edge deep reinforcement learning techniques in terms of convergence speed, battery life extension, fuel consumption, and overall driving cost reduction. The proposed control strategy can improve the convergence speed by nearly 50%, while effectively extending the battery life and reducing the overall driving cost compared to the existing state-of-the-art strategies. The battery degradation rate is reduced by 48.46%, 57.95%, and 36.99%, and the driving cost is reduced by 17.76%, 8.51%, and 7.12%, respectively, under each usage scenario.


---

## Body

## 1. Introduction

The heavy-duty truck (HDT) class is one of the hard-to-replace transportation sectors [1]. Freight shipments are increasing worldwide due to globalization [2, 3], while trucks generate a disproportionate amount of environmental pollution [4, 5]. With the pressure of energy shortage and environmental degradation, there is a growing interest in reducing fuel consumption for heavy-duty trucks in response to decarbonization [6, 7]. Plug-in hybrid electric vehicles (PHEVs) are known as one of the most guaranteed possibilities for energy-saving and emission reduction. PHEVs have been widely concerned and studied in the heavy-duty truck sector for the major advantages of fuel efficiency and range extension.Energy management strategies (EMSs) play an essential duty in the energy-saving optimization of plug-in hybrid electric heavy-duty trucks (PHETs) by integrating the overall efficiency and energy circulation of the battery, engine, and motor. Researchers have carried out substantial studies on different types of energy management control strategies. Up-to-date energy management strategies can be broadly classified into rule-based and optimization-based [8]. The rule-based strategies are broadly applicable to real vehicle control, which is relatively easy to implement. However, such empirical strategies cannot guarantee optimality when encountering the phenomenon of incompatible parameter settings. Strategies such as dynamic programming (DP) [9], Pontryagin’s minimum principle (PMP) [10], equivalent consumption minimization strategy (ECMS) [11], and model predictive control (MPC) [12, 13] are some of the mainstream optimization-based strategies. The DP algorithm is the most representative global optimization algorithm and is generally deemed as a benchmark due to its globally optimal performance. But it cannot be practically applied because of its huge computational cost. PMP is used for solving the optimal control problem with constrained control action, so a more precise cooperative state prediction model is required. However, it is not feasible to achieve real-time control due to its great computational effort. ECMS is an algorithm for real-time application. Despite the real-time applicability of ECMS, its sensitivity to driving conditions makes it necessary to consider the effect of future operating conditions on the equivalent factor. MPC combines instantaneous and global optimization, providing good dynamic control performance and robustness. However, a well-optimized control time domain and prediction duration need to be well-selected to ensure the optimality of MPC.The development of artificial intelligence techniques makes a spurt of progress [14]. The deep reinforcement learning (DRL) technique has become a practical and essential approach for developing model-free energy management strategies. Recent researchers apply algorithms directly to EMSs, such as deep Q-learning (DQL) [15, 16], double deep Q-learning (DDQN) [17], and deep deterministic policy gradient (DDPG) [18, 19], which enables match from multidimensional state-aware inputs to optimum action outputs. Hybrid algorithms are progressively developed simultaneously [20], which combine other information or algorithms with reinforcement learning (RL), such as predictive algorithms, trip information, deep learning, and MPC. Li et al. [21] integrated real-time topographic information into a DRL-based EMS, which exhibited a superior control effect compared to conventional DRL. Wu et al. [22] incorporated traffic information and the variety of passengers into the EMS, using a variety of driving cycles generated by traffic simulation for training to obtain a performance close to DP.Also, in recent years, integrating RL with transfer learning (TL) has been an appealing approach to solving vehicle control problems. TL is yet another learning prototype in machine learning techniques [23] that is aimed at solving challenging learning problems with few or even no labeled samples in the target domain. Once a trained model has been acquired in one domain, this model can be introduced into a similar domain, thereby accelerating the learning process. As mentioned earlier, the DRL technique solves some of the critical problems of EMS to a certain extent, but there are still some drawbacks. The agent always needs to find the optimal solution through exploration and trial in interacting with the environment. It takes more time and requires retraining even for EMS problems in a similar domain. Therefore, a fraction of researchers have studied TL in combination with DRL-based EMS to address this problem. Lian et al. [24] applied TL technology to transfer knowledge of DRL-based EMS among vehicles with four different topologies. The impact of adding noise on the convergence speed of deep transfer learning has been investigated by Xu et al. [25], and they find that adding noise to the parameter space can increase the robustness and obtain a faster convergence speed. To further realize the real-time application of EMS, Guo et al. [26] constructed a two-layer control framework for EMS. The upper layer applied the DDPG algorithm to train the EMS for distinct velocity intervals, and the bottom layer used the TL technique to transform the pretrained neural network into a real application. Furthermore, He et al. [27] combined information-physical systems with deep transfer learning (DTL) and transferred energy management strategies between different types of vehicles.Although RL is commonly used in EMS, three main problems still need to be improved. Firstly, existing energy management strategies are tested under standard driving cycles, without considering the real usage scenarios of vehicles. Secondly, the trained control strategy, which is optimal in one particular driving cycle and suboptimal in another [28], cannot guarantee all-time optimality for energy management. Thirdly, when RL is faced with a completely new driving cycle, it needs to reexplore the learning, which will be time-consuming.Also, considering that PHET is one form of plug-in hybrid electric vehicle. Due to the unique transportation nature of PHETs, which often drive in scenarios with different characteristics, it is difficult to make an energy management strategy with complex scenario characteristics, while driving pattern recognition (DPR) can solve the energy management problem of variable usage scenarios. Wei et al. [29] combined variational mode decomposition (VMD) and extreme learning machine (ELM) to design a novel driving pattern recognition approach, while proposing an adaptive equivalent consumption minimization strategy for extender range electric logistics vehicles to improve fuel consumption. Shi et al. [30] incorporated vehicle-to-cloud (V2C) information into DPR-based energy management to reduce battery degradation and energy consumption cost. Tian et al. [31] proposed an adaptive energy management system consisting of offline and online components, in which the offline component accurately identifies the driver’s driving style and the online component combines the identification results with the ECMS-based energy management to effectively improve both charging sustainability and equivalent fuel consumption. Song et al. [32] proposed a multimode energy management strategy based on DPR technology. The multimode control strategy is switched according to the vehicle DPR identifier. The results show that the economic performance is effectively improved under dynamic driving cycles. Guo et al. [33] analyzed the accelerator pedal opening and its change rate under different driving cycles and established a fuzzy logic recognizer for DPR. Then, the identified driving styles were combined with ECMS to achieve adaptive optimal control.The main objective of this paper is to build an EMS that can cope well with the complex usage scenarios of PHETs. It is also aimed at minimizing the overall driving cost and at solving the problems of slow convergence and poor generalization of control strategies based on DRL. Therefore, to bridge the abovementioned gap, a deep transfer reinforcement learning- (DTRL-) based EMS is proposed. To the best of our knowledge, is no efforts have been made to combine the DDPG algorithm with TL technique, with the assistance of DPR for energy management that provides new ideas for the practical application of EMS for PHETs. The paper contains the following four perspectives that would possibly contribute:(1)
Representative driving cycles are constructed based on actual PHETs’ history data in different usage scenarios, which can reflect the real driving characteristics of the PHET in the usage scenario. Representative driving cycles are used as the training set for DPR and EMS source domains(2)
Driving pattern recognition based on learning vector quantization (LVQ) is proposed to provide accurate usage scenario recognition of the real driving cycles. The accurate identification of DPR helps to guarantee the implementation of transfer tasks(3)
Within the framework of DDPG-based EMS, combined with TL, faster convergence and better control performance are achieved in different real usage scenarios(4)
The battery aging cost is considered, and the proposed strategy’s supremacy is authenticated through detailed comparisons among state-of-the-art EMSs, primarily DQN-based and DP-based strategiesTo the best of our knowledge, there are no efforts that have been made to combine the DDPG algorithm with TL technique, with the assistance of DPR for energy management.The rest of the paper proceeds as follows. The PHET powertrain model is introduced in Section2. A naturalistic driving data-driven approach is used to construct three distinct driving cycles in Section 3. Section 4 contains an LVQ-based DPR to deal with driving pattern recognition, and a DDPG-based EMS with TL technique to dispose of the control problem is formulated. Simulation results are analyzed in Section 5, followed by significant conclusions summarized in Section 6.


## 2. Powertrain Configuration

The powertrain system of the considered plug-in hybrid electric heavy-duty truck (PHET) mainly includes the engine, motor generators (MGs), and planetary gear disk. The power-split configuration is used to accomplish dual decoupling of engine speed and torque, resulting in higher engine operating efficiency, thus becoming the mainstream configuration of PHET. The analysis situs of PHET’s powertrain is shown in Figure1. A widely used quasistatic model is used for modeling dynamic systems. The specific parameters are listed in Table 1.Figure 1
The analysis situs of the PHET under investigation.Table 1
Key specifications of the studied PHET.
ComponentParametersValuesDiesel engineMaximum power/torque172 kW/756 nmMaximum speed2300 rpmDriving motor (MG1)Maximum power/torque187 kW/658 nmMaximum speed1570 rpmElectricity-generating motor (MG2)Maximum power/torque167 kW/574 nmMaximum speed1519 rpmBatteryTypeLIBNominal energy27.232kWhNominal voltage537.6 VTransmissionRatio of planetary gear2.60Ratio of single reducer2.636Ratio of final single gear4.364Vehicle dynamicCurb weight17000 kgWheel radius0.45 mAir resistance coefficient0.7Air density1.225 kg/m3Frontal area5.2 m2Road resistance coefficient0.01


### 2.1. Vehicle Dynamics Modeling

The powertrain provides the demand tractive force of the vehicle, which can be calculated based on longitudinal vehicular dynamics as(1)Ft=Gf+CDAρv2/2+δmdvdt,where G is the gravity acceleration, f is the rolling resistance coefficient, CD is the air resistance coefficient, A is the frontal area, ρ is the air density, v is the vehicle relative vehicle speed, δ is the rotating mass conversion factor, and m is the vehicle mass.Figure2 illustrates the powertrain’s specific mechanical structure. The planetary gear (PG) functions as a continuously variable transmission, which can deconstruct the engine speed from vehicle speed. The output torque is coupled at the planetary frame of PG2 and transmitted to wheels through the main gear reducer. The coupling relationship between the powertrain components is described as
(2)TMG1:Teng:TMG2=1:−1+k1:k1k2,nMG1+k1nMG2k2−1+αneng=0,where TMG1, Teng, and TMG2 are the torque of the generator, engine, and motor, respectively; nMG1, nMG2, and neng are the speed of the generator, engine, and motor, respectively; and k1 and k2 are the gear ratio of the planetary gear and single reducer, respectively. The output torque of the transmission and the required traction force can be described as
(3)Tt=α1+αTeng+imTMG2,Ft=Tt×i0Rw,where Tt is the output torque of transmission, i0 is the final gear ratio, and Rw is the wheel radius.Figure 2
Mechanical structure of the PHET powertrain.


### 2.2. Power Component Modeling

The static models of the engine and motor are constructed in this paper. The fuel consumption concluded through benchmark tests at different combinations of speed and torque, which is given as(4)mfuel=Peng×bengTeng,weng3.6×106,Peng=Teng×weng,where mfuel is the engine fuel consumption per unit time and bengTeng,weng is the fuel consumption rate. Also, the actual power demand of motor operating in two different modes (motor mode and generator mode) is expressed as follows:
(5)Pi=Tmωmηm,motormodeTgωgηg,generatormode,where Tm and ωm are the motor torque and speed, Tg and ωg are the generator torque and speed, and ηm and ηg are the motor efficiency and generator efficiency, respectively.


### 2.3. Battery Circuit Model

The first-order equivalent internal resistance model is used in this paper, and the battery unit can be calculated from the following internal resistance model:(6)Vbatt=ut−ItRbat,(7)Pbatt=VbattIt,where Vbat, u, I, Rbat, and Pbat represent the terminal voltage, open-circuit voltage, current, internal resistance of battery cell, and battery power, respectively.The current equation can be further calculated from (6) and (7) as follows:
(8)It=ut−ut2−4RbatPbatt2R.The state of charge (SoC) of battery can then be calculated by(9)SoCt+1=SoCt−ItΔtQ,where SoC, Q, and Δt represent the battery state of charge, the nominal capacity of the battery, and the sampling interval, respectively. The SoC is commonly used to represent the current energy stored in the battery, which is an individual state variable in this paper.


### 2.4. Battery Aging Model

A semiempirical model proposed by [31] is adopted in this paper to calculate battery degradation. This model uses a throughput-based capacity degradation model, which assumes that the battery can withstand a certain amount of accumulated charge before the end of its life. Thus, the state of health (SoH) can be characterized by the proportion of accumulated charge consumed.The computational process of the model is further described next, where first, the capacity loss of the battery can be calculated by the following:(10)ΔQ=Bc×e−Eac/RTAc,Taz,where ΔQ is the percentage of capacity loss, B is the preexponential factor of the C-rate (refer to Table 2), c is the C-rate, R is the ideal gas constant, T is the lumped battery temperature, and A is the discharged ampere-hour throughput. The A123’s LiFePO4 (ANR26650M1) battery model is used in this paper. More exact values of model parameters can be referred to in [34]. The activation energy Ea and the power-law factor z are defined by
(11)Eac=31700−370.3×c,z=0.55.Table 2
Dependence of pre-exponential factor toC-rate.
c0.52610Bc31630216811293415512Ebbesen et al. [35] concluded that the main variables affecting battery capacity decay are related to C-rate and ambient temperature. This paper assumes that a thermal management system is used to keep a constant total battery temperature, so C-rate is the only time-varying variable that appreciably affects the battery life. Therefore, the main obstacle is obtaining a thoughtful expression representing the effect of C-rate on cycle number N. Also, the end of battery life is usually defined as when the battery capacity decreases by 20%. Thus, the total discharged Ah throughput is hence calculated by
(12)Atotalc,Ta=20/Bc×exp−EacRTa1/z.The number of cycles of the battery before the end of life is expressed as follows:(13)Nc,Ta=3600Atotalc,TaQ,where N is the equivalent number of cycles till the end of life; eventually, combined with (12), the battery health state is calculated from the following equation:
(14)SoHt=1−∫0tItdτ2Nc,TaQ.The dynamics of the time discretization to calculate SOH can be represented as follows:(15)SoHt+1=SoHt−ItΔt2Nc,TaQ.


## 2.1. Vehicle Dynamics Modeling

The powertrain provides the demand tractive force of the vehicle, which can be calculated based on longitudinal vehicular dynamics as(1)Ft=Gf+CDAρv2/2+δmdvdt,where G is the gravity acceleration, f is the rolling resistance coefficient, CD is the air resistance coefficient, A is the frontal area, ρ is the air density, v is the vehicle relative vehicle speed, δ is the rotating mass conversion factor, and m is the vehicle mass.Figure2 illustrates the powertrain’s specific mechanical structure. The planetary gear (PG) functions as a continuously variable transmission, which can deconstruct the engine speed from vehicle speed. The output torque is coupled at the planetary frame of PG2 and transmitted to wheels through the main gear reducer. The coupling relationship between the powertrain components is described as
(2)TMG1:Teng:TMG2=1:−1+k1:k1k2,nMG1+k1nMG2k2−1+αneng=0,where TMG1, Teng, and TMG2 are the torque of the generator, engine, and motor, respectively; nMG1, nMG2, and neng are the speed of the generator, engine, and motor, respectively; and k1 and k2 are the gear ratio of the planetary gear and single reducer, respectively. The output torque of the transmission and the required traction force can be described as
(3)Tt=α1+αTeng+imTMG2,Ft=Tt×i0Rw,where Tt is the output torque of transmission, i0 is the final gear ratio, and Rw is the wheel radius.Figure 2
Mechanical structure of the PHET powertrain.


## 2.2. Power Component Modeling

The static models of the engine and motor are constructed in this paper. The fuel consumption concluded through benchmark tests at different combinations of speed and torque, which is given as(4)mfuel=Peng×bengTeng,weng3.6×106,Peng=Teng×weng,where mfuel is the engine fuel consumption per unit time and bengTeng,weng is the fuel consumption rate. Also, the actual power demand of motor operating in two different modes (motor mode and generator mode) is expressed as follows:
(5)Pi=Tmωmηm,motormodeTgωgηg,generatormode,where Tm and ωm are the motor torque and speed, Tg and ωg are the generator torque and speed, and ηm and ηg are the motor efficiency and generator efficiency, respectively.


## 2.3. Battery Circuit Model

The first-order equivalent internal resistance model is used in this paper, and the battery unit can be calculated from the following internal resistance model:(6)Vbatt=ut−ItRbat,(7)Pbatt=VbattIt,where Vbat, u, I, Rbat, and Pbat represent the terminal voltage, open-circuit voltage, current, internal resistance of battery cell, and battery power, respectively.The current equation can be further calculated from (6) and (7) as follows:
(8)It=ut−ut2−4RbatPbatt2R.The state of charge (SoC) of battery can then be calculated by(9)SoCt+1=SoCt−ItΔtQ,where SoC, Q, and Δt represent the battery state of charge, the nominal capacity of the battery, and the sampling interval, respectively. The SoC is commonly used to represent the current energy stored in the battery, which is an individual state variable in this paper.


## 2.4. Battery Aging Model

A semiempirical model proposed by [31] is adopted in this paper to calculate battery degradation. This model uses a throughput-based capacity degradation model, which assumes that the battery can withstand a certain amount of accumulated charge before the end of its life. Thus, the state of health (SoH) can be characterized by the proportion of accumulated charge consumed.The computational process of the model is further described next, where first, the capacity loss of the battery can be calculated by the following:(10)ΔQ=Bc×e−Eac/RTAc,Taz,where ΔQ is the percentage of capacity loss, B is the preexponential factor of the C-rate (refer to Table 2), c is the C-rate, R is the ideal gas constant, T is the lumped battery temperature, and A is the discharged ampere-hour throughput. The A123’s LiFePO4 (ANR26650M1) battery model is used in this paper. More exact values of model parameters can be referred to in [34]. The activation energy Ea and the power-law factor z are defined by
(11)Eac=31700−370.3×c,z=0.55.Table 2
Dependence of pre-exponential factor toC-rate.
c0.52610Bc31630216811293415512Ebbesen et al. [35] concluded that the main variables affecting battery capacity decay are related to C-rate and ambient temperature. This paper assumes that a thermal management system is used to keep a constant total battery temperature, so C-rate is the only time-varying variable that appreciably affects the battery life. Therefore, the main obstacle is obtaining a thoughtful expression representing the effect of C-rate on cycle number N. Also, the end of battery life is usually defined as when the battery capacity decreases by 20%. Thus, the total discharged Ah throughput is hence calculated by
(12)Atotalc,Ta=20/Bc×exp−EacRTa1/z.The number of cycles of the battery before the end of life is expressed as follows:(13)Nc,Ta=3600Atotalc,TaQ,where N is the equivalent number of cycles till the end of life; eventually, combined with (12), the battery health state is calculated from the following equation:
(14)SoHt=1−∫0tItdτ2Nc,TaQ.The dynamics of the time discretization to calculate SOH can be represented as follows:(15)SoHt+1=SoHt−ItΔt2Nc,TaQ.


## 3. Driving Cycle Construction and Recognition under Segmented Usage Scenarios

The energy management of most PHETs is based on standard driving cycles. Due to the unique nature of transportation tasks, the standard driving cycles are not applicable. Trucks have different driving characteristics when PHETs are driving in various usage scenarios. It is of great worthwhile relevance to collect the real driving data of PHETs and establish typical driving cycles.


### 3.1. Driving Cycle Construction of PHET

The real driving data of PHETs in different usage scenarios are collected using cloud-based big data. The usage scenarios of PHETs in this paper are mainly divided into urban construction dump transport, mining transport, and coal transport. PHETs upload their data to the cloud platform every second with the assistance of on-board diagnostics (OBD) and vehicle-to-X (V2X) systems. A total of 1150 trucks were collected in this paper, including 417 urban construction dump transport trucks, 200 mining transport trucks, and 534 coal transport trucks. More than 360 million seconds of driving information are collected. Partially collected speed trajectories of PHETs are shown in Figure3.Figure 3
Collected speed trajectories of PHETs.Data processing is used as the primary step in building the driving cycle. During the collection process, the raw data are often subject to burrs or abrupt changes due to the factors such as driving environment, interference signals, and buildings. Therefore, noise reduction and smoothing processes are performed on the raw data to avoid disturbing factors in the driving environment. Wavelet decomposition and reconstruction are used to improve the signal’s distortion, noise, and corruption. Noise and real signals have different wavelet decomposition coefficients, and the wavelet denoising technique is based on this principle by eliminating wavelet coefficients below a set threshold and reconstructing them to obtain clean data. The preliminary and preprocessed data are shown in Figure4. Comparing the preliminary data with the preprocessed data, it can be concluded that the speed curve becomes smoother after the denoising process. The peaks are eliminated, which significantly improves the signal-to-noise ratio and ensures that the driving cycles are constructed using valid data.Figure 4
Comparison between the preliminary and preprocessed data.Next, kinematic segmentation was performed on the preprocessed data. The kinematic segments were described as follows:(16)v<2&absa<0.15,idling,absa≥0.15,acceleration,a≤−0.15,decelecration,v≥2&absa<0.15,crusing,where v (km/h) and a (m/s2) are the speed and acceleration of the PHET, respectively.After segmenting the preprocessed data, suitable parameters were chosen to characterize the segments. Using all the parameters will result in too many feature parameters and more complicated analysis. If all the kinematic vehicle information is retained, there will be many parameters with strong correlation, so the dimensions of feature parameters were reduced using the principal component analysis (PCA) algorithm.Therefore, this paper uses PCA algorithm to reduce the dimensionality of the kinematic feature parameters. First, ten kinematic feature parameters are chosen to assemble new complete evaluation indexes, as proven in (17): maximum speed, minimum speed, average speed, standard deviation of speed, maximum acceleration, maximum deceleration, average acceleration, standard deviation of acceleration, speed difference, and running time.
(17)y1=a11x1+a12x2+⋯+a110x10,y2=a21x1+a22x2+⋯+a210x10,⋯y10=a110x1+a102x2+⋯+a1010x10,where x1,x2⋯x10 are the normalized kinematic characteristic parameter; y1,y2⋯y10 are the ten composite indexes; and a1,a2⋯a10 are the kinematic characteristic parameter coefficient. After further orthogonal transformation, the expressions of principal components are as follows:
(18)y1′=l1TX=l11x1+l12x2+⋯+l110x10,y2′=l2TX=l21x1+l22x2+⋯+l210x10,⋯y10′=l10TX=l110x1+l102x2+⋯+l1010x10,where y1′,y2′⋯y10′ are mutually independent called the first, second, and tenth principal components, respectively, l1,l2⋯l10 and whether or not the unit eigenvectors correspond to the descending order of the eigenvalues λ1,λ2⋯λ10 (λ1≥λ2≥⋯≥λ10).The PCA results of the feature parameters are shown in Table3. The PCA selected the first four principal components, whose cumulative contribution rate exceeded 90%. At the same time, it can be shown that the selected characteristics after dimensionality reduction processing by PCA have strong noncorrelation and can reflect the original data information well.Table 3
The results of principal component analysis.
Principal componentVarianceContribution rate (%)Cumulative contribution rate (%)15.576255.76255.76222.021820.22875.99031.011210.11286.10240.79137.91394.01550.39753.97597.99The data after preprocessing and principal component analysis need to be clustered to construct the driving cycles. In the current literature, theK-means clustering algorithm has become the most used algorithm because of its computational simplicity. However, the K-means clustering algorithm also has obvious disadvantages. K-means is a hard clustering algorithm, which uses the Euclidean distance from the data to the cluster center as a classification criterion. When clustering, the initial K-value is difficult to choose, and also because the iterative update algorithm is adopted, it easily falls into the local optimum. To remedy this drawback of the K-means algorithm, this paper uses a support vector machine (SVM) and K-means hybrid classification algorithm for clustering analysis.Then, representative driving cycles for three usage scenarios were going to be constructed from the data after clustering analysis. Considering the dynamic characteristics of the vehicles and the time series of the driving cycles, the Markov-Monte Carlo simulation method is used to construct driving cycles, to preserve finely resolved driving variability and microscopic features. The flow chart of the Markov-Monte Carlo simulation method is shown in Figure5.Figure 5
Flowchart of Markov and Monte Carlo simulation method.Specific details during the construction of driving cycles, such as the selection of training set, classification process in support vector machine basedK-means (SVM-K-means), and Markov transfer probability matrix, have been discussed in our previous study [36]. The duration of the representative driving cycle constructed under each usage scenario is 1800 seconds, as shown in Figure 6. The urban construction dump transportation scenario, coal transportation scenario, and mining transportation scenario are denoted as scenario A, scenario B, and scenario C, respectively, as shown in Table 4.Figure 6
Representative driving cycles. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)Table 4
Segmented usage scenarios.
Usage scenarioUrban construction dump transportCoal transportMining transportNumberScenario AScenario BScenario C


### 3.2. The Establishment of the Driving Pattern Recognition Model

Driving pattern recognition (DPR), in which feature parameters are sampled and extracted from previous driving cycles, uses a well-trained classifier to recognize the current driving pattern. Learning vector quantization (LVQ) is a simple structured and powerful supervised neural network classification method. As a kind of classification method, LVQ has been effectively applied in statistics, pattern recognition [37, 38], etc. Therefore, this paper selects LVQ as a driving pattern recognizer for DPR, and its structure is shown in Figure 7.Figure 7
The LVQ neural network structure.The LVQ neural network has two main layers: the competitive and linear layers. In the competitive layer, the network combines competitive learning and supervises learning to classify the input vectorX. Within the neuron, the competitive layer selects the best coordination neuron and adaptively updates the weight vector. Then, the linear layer passes the classification results of the competing layer to the user-defined target classification.Meanwhile, based on PHTEs’ driving characteristics, to accurately identify the working conditions, 10 characteristic parameters are selected to be the input of the LVQ network. These 10 characteristic parameters are selected by calculating the Pearson correlation coefficients between classical characteristic parameters, which are shown in Table5.Table 5
Characteristic parameters of typically synthetic driving cycles.
Assessment parameterScenario AScenario BScenario CMaximum speed (km/h)64.8765.0286.00Average speed (km/h)18.7813.8041.74Standard deviation of speed21.1315.7027.97Average acceleration (m/s2)0.310.380.30Average deceleration (m/s2)-0.42-0.45-0.46Standard deviation of acceleration0.230.270.28Percentage of acceleration (%)0.180.170.25Percentage of deceleration (%)0.120.150.17Percentage of constant speed (%)0.300.270.12Percentage of idling (%)0.390.400.44This paper employs a data-driven method for DPR to realize the 3 usage scenarios recognition. Firstly, due to the limited duration of the constructed representative driving cycles, one single driving cycle is not enough to provide enough training samples. Each of the driving cycles established in the previous paper is repeated 10 times in series as a longer driving cycle, and the total length becomes 18000 s.Secondly, in determining the training set, this paper uses the sliding window approach for parameter extraction to increase the number of training samples. The length of the sliding window is selected as 1800 s, with 100 s as the interval time. The characteristic parameters are extracted from the speed interval in the window and input to the recognizer for training. The data was divided into 742 samples. 80 percent were selected as the training set, and 20 percent as the test set. According to the above design scheme, the accuracy of the trained recognizer designed in this paper is 95.8%.Finally, when using the recognizer to the real driving data, according to the previous research, 25 s was selected as the initial recognition horizon. Then, we extract feature parameters for DPR judgment on all historical vehicle speeds every 25 second interval. This is done because the characteristic parameters selected at the beginning may not reflect the characteristics of the entire driving cycle due to their short duration. With the increase of the accumulated historical time, the characteristics of the driving cycles become more representative of the usage scenario in which the PHET is operating.The correct identification of the usage scenario to which the vehicle belongs can provide a guarantee for the transfer learning-based energy management framework in the later section. Three constructed representative driving cycles are selected as the source conditions. The DRL algorithm focuses on training the neural network parameters in each representative driving cycle. Then, the trained neural network will be transferred to the real driving data to which it belongs to obtain higher efficiency and control effect. The DDPG algorithm and DTL algorithm are introduced in the next section.


## 3.1. Driving Cycle Construction of PHET

The real driving data of PHETs in different usage scenarios are collected using cloud-based big data. The usage scenarios of PHETs in this paper are mainly divided into urban construction dump transport, mining transport, and coal transport. PHETs upload their data to the cloud platform every second with the assistance of on-board diagnostics (OBD) and vehicle-to-X (V2X) systems. A total of 1150 trucks were collected in this paper, including 417 urban construction dump transport trucks, 200 mining transport trucks, and 534 coal transport trucks. More than 360 million seconds of driving information are collected. Partially collected speed trajectories of PHETs are shown in Figure3.Figure 3
Collected speed trajectories of PHETs.Data processing is used as the primary step in building the driving cycle. During the collection process, the raw data are often subject to burrs or abrupt changes due to the factors such as driving environment, interference signals, and buildings. Therefore, noise reduction and smoothing processes are performed on the raw data to avoid disturbing factors in the driving environment. Wavelet decomposition and reconstruction are used to improve the signal’s distortion, noise, and corruption. Noise and real signals have different wavelet decomposition coefficients, and the wavelet denoising technique is based on this principle by eliminating wavelet coefficients below a set threshold and reconstructing them to obtain clean data. The preliminary and preprocessed data are shown in Figure4. Comparing the preliminary data with the preprocessed data, it can be concluded that the speed curve becomes smoother after the denoising process. The peaks are eliminated, which significantly improves the signal-to-noise ratio and ensures that the driving cycles are constructed using valid data.Figure 4
Comparison between the preliminary and preprocessed data.Next, kinematic segmentation was performed on the preprocessed data. The kinematic segments were described as follows:(16)v<2&absa<0.15,idling,absa≥0.15,acceleration,a≤−0.15,decelecration,v≥2&absa<0.15,crusing,where v (km/h) and a (m/s2) are the speed and acceleration of the PHET, respectively.After segmenting the preprocessed data, suitable parameters were chosen to characterize the segments. Using all the parameters will result in too many feature parameters and more complicated analysis. If all the kinematic vehicle information is retained, there will be many parameters with strong correlation, so the dimensions of feature parameters were reduced using the principal component analysis (PCA) algorithm.Therefore, this paper uses PCA algorithm to reduce the dimensionality of the kinematic feature parameters. First, ten kinematic feature parameters are chosen to assemble new complete evaluation indexes, as proven in (17): maximum speed, minimum speed, average speed, standard deviation of speed, maximum acceleration, maximum deceleration, average acceleration, standard deviation of acceleration, speed difference, and running time.
(17)y1=a11x1+a12x2+⋯+a110x10,y2=a21x1+a22x2+⋯+a210x10,⋯y10=a110x1+a102x2+⋯+a1010x10,where x1,x2⋯x10 are the normalized kinematic characteristic parameter; y1,y2⋯y10 are the ten composite indexes; and a1,a2⋯a10 are the kinematic characteristic parameter coefficient. After further orthogonal transformation, the expressions of principal components are as follows:
(18)y1′=l1TX=l11x1+l12x2+⋯+l110x10,y2′=l2TX=l21x1+l22x2+⋯+l210x10,⋯y10′=l10TX=l110x1+l102x2+⋯+l1010x10,where y1′,y2′⋯y10′ are mutually independent called the first, second, and tenth principal components, respectively, l1,l2⋯l10 and whether or not the unit eigenvectors correspond to the descending order of the eigenvalues λ1,λ2⋯λ10 (λ1≥λ2≥⋯≥λ10).The PCA results of the feature parameters are shown in Table3. The PCA selected the first four principal components, whose cumulative contribution rate exceeded 90%. At the same time, it can be shown that the selected characteristics after dimensionality reduction processing by PCA have strong noncorrelation and can reflect the original data information well.Table 3
The results of principal component analysis.
Principal componentVarianceContribution rate (%)Cumulative contribution rate (%)15.576255.76255.76222.021820.22875.99031.011210.11286.10240.79137.91394.01550.39753.97597.99The data after preprocessing and principal component analysis need to be clustered to construct the driving cycles. In the current literature, theK-means clustering algorithm has become the most used algorithm because of its computational simplicity. However, the K-means clustering algorithm also has obvious disadvantages. K-means is a hard clustering algorithm, which uses the Euclidean distance from the data to the cluster center as a classification criterion. When clustering, the initial K-value is difficult to choose, and also because the iterative update algorithm is adopted, it easily falls into the local optimum. To remedy this drawback of the K-means algorithm, this paper uses a support vector machine (SVM) and K-means hybrid classification algorithm for clustering analysis.Then, representative driving cycles for three usage scenarios were going to be constructed from the data after clustering analysis. Considering the dynamic characteristics of the vehicles and the time series of the driving cycles, the Markov-Monte Carlo simulation method is used to construct driving cycles, to preserve finely resolved driving variability and microscopic features. The flow chart of the Markov-Monte Carlo simulation method is shown in Figure5.Figure 5
Flowchart of Markov and Monte Carlo simulation method.Specific details during the construction of driving cycles, such as the selection of training set, classification process in support vector machine basedK-means (SVM-K-means), and Markov transfer probability matrix, have been discussed in our previous study [36]. The duration of the representative driving cycle constructed under each usage scenario is 1800 seconds, as shown in Figure 6. The urban construction dump transportation scenario, coal transportation scenario, and mining transportation scenario are denoted as scenario A, scenario B, and scenario C, respectively, as shown in Table 4.Figure 6
Representative driving cycles. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)Table 4
Segmented usage scenarios.
Usage scenarioUrban construction dump transportCoal transportMining transportNumberScenario AScenario BScenario C


## 3.2. The Establishment of the Driving Pattern Recognition Model

Driving pattern recognition (DPR), in which feature parameters are sampled and extracted from previous driving cycles, uses a well-trained classifier to recognize the current driving pattern. Learning vector quantization (LVQ) is a simple structured and powerful supervised neural network classification method. As a kind of classification method, LVQ has been effectively applied in statistics, pattern recognition [37, 38], etc. Therefore, this paper selects LVQ as a driving pattern recognizer for DPR, and its structure is shown in Figure 7.Figure 7
The LVQ neural network structure.The LVQ neural network has two main layers: the competitive and linear layers. In the competitive layer, the network combines competitive learning and supervises learning to classify the input vectorX. Within the neuron, the competitive layer selects the best coordination neuron and adaptively updates the weight vector. Then, the linear layer passes the classification results of the competing layer to the user-defined target classification.Meanwhile, based on PHTEs’ driving characteristics, to accurately identify the working conditions, 10 characteristic parameters are selected to be the input of the LVQ network. These 10 characteristic parameters are selected by calculating the Pearson correlation coefficients between classical characteristic parameters, which are shown in Table5.Table 5
Characteristic parameters of typically synthetic driving cycles.
Assessment parameterScenario AScenario BScenario CMaximum speed (km/h)64.8765.0286.00Average speed (km/h)18.7813.8041.74Standard deviation of speed21.1315.7027.97Average acceleration (m/s2)0.310.380.30Average deceleration (m/s2)-0.42-0.45-0.46Standard deviation of acceleration0.230.270.28Percentage of acceleration (%)0.180.170.25Percentage of deceleration (%)0.120.150.17Percentage of constant speed (%)0.300.270.12Percentage of idling (%)0.390.400.44This paper employs a data-driven method for DPR to realize the 3 usage scenarios recognition. Firstly, due to the limited duration of the constructed representative driving cycles, one single driving cycle is not enough to provide enough training samples. Each of the driving cycles established in the previous paper is repeated 10 times in series as a longer driving cycle, and the total length becomes 18000 s.Secondly, in determining the training set, this paper uses the sliding window approach for parameter extraction to increase the number of training samples. The length of the sliding window is selected as 1800 s, with 100 s as the interval time. The characteristic parameters are extracted from the speed interval in the window and input to the recognizer for training. The data was divided into 742 samples. 80 percent were selected as the training set, and 20 percent as the test set. According to the above design scheme, the accuracy of the trained recognizer designed in this paper is 95.8%.Finally, when using the recognizer to the real driving data, according to the previous research, 25 s was selected as the initial recognition horizon. Then, we extract feature parameters for DPR judgment on all historical vehicle speeds every 25 second interval. This is done because the characteristic parameters selected at the beginning may not reflect the characteristics of the entire driving cycle due to their short duration. With the increase of the accumulated historical time, the characteristics of the driving cycles become more representative of the usage scenario in which the PHET is operating.The correct identification of the usage scenario to which the vehicle belongs can provide a guarantee for the transfer learning-based energy management framework in the later section. Three constructed representative driving cycles are selected as the source conditions. The DRL algorithm focuses on training the neural network parameters in each representative driving cycle. Then, the trained neural network will be transferred to the real driving data to which it belongs to obtain higher efficiency and control effect. The DDPG algorithm and DTL algorithm are introduced in the next section.


## 4. DRL-Based Energy Management Framework

### 4.1. The Formulation of DDPG Algorithm

RL is a machine learning technique that learns from the interplay between agent and environment, which derives the optimal control policy by trial-and-error methods in performing the behavior and getting feedback from the environment. In the process of reinforcement learning, the agent updates and learns the strategy by obtaining the goal of maximum cumulative reward, and the accumulative discounted expected reward is(19)Rt=∑i=tTγi−trsi,ai,where t is the time step, γ is a discount factor of size 0 to 1 to adjust the importance between immediate and future rewards, and r is the immediate reward. The optimum control strategy learned by reinforcement learning gives the mapping from state to action. Obtaining the maximum cumulative reward is only the means, and finding the method to solve the optimal control problem is the ultimate goal.Recent years have witnessed a spurt of machine learning progress, while engineering control problems have become increasingly complex. DDPG was created to manage problems in multidimensional continuous state and action spaces [39]. DDPG is an off-policy, off-model DRL, a deep reinforcement learning algorithm developed based on an actor-critic architecture. Actor-network μsθμ is input with state observations, which are then mapped to a deterministic action by the neural network. Critical-network Qsθμ is input with observations of the action taken by the actor-network and the current state to evaluate the merit of the present action.In addition, the experience replay buffer is introduced into the DDPG to reduce the randomness and dependence among samples, which can also reduce the bias arising from the estimation of the value function. However, to minimize the bias arising from taking a single critic forQ-value estimation and a single actor-network for action selection, target actor-network μ′sθμ′ and target critic-network Q′sθQ′ are introduced to estimate the Q-value:
(20)yt=rt+γQ′st+1,μ′st+1θμ′θQ′,where L is the average loss and N is the fixed size of the minibatch, arbitrarily chosen from the empirical replay buffer. The selection action by the actor-network μsθμ is to maximize the Q-value. Thus, the parameter update θμ can be solved numerically by applying the gradient method. The derived chain rule is applied as follows:
(21)∇θμJ≈1N∑i∇aQsi,aiθQ∇θμμsiθμ.Meanwhile, the target networksμ′ and Q′ using time lag updates can significantly improve the stability of learning:
(22)θQ′⟵τθQ+1−τθQ′,θμ′⟵τθμ+1−τθμ′,where τ is the soft update factor and θ and θ′ are the original network and target network parameters, respectively. The DDPG algorithm also improves the training efficiency by using priority experience replay to enhance the training process’s stability and the model’s robustness, while extra noise is introduced to the output of the actor-network to allow the DDPG agent to perform better exploration and select the correct action. In this paper, softmax action noise (SAN) noise is employed.


### 4.2. Energy Management Problem

The core opinion of EMS is to transform the energy management problem into an optimal control problem with distinct optimization objectives and control restrictions. EMS needs to make a reasonable allocation of power under the premise of satisfying vehicle demand power. After each power allocation decision, the driving environment, vehicle state switch, and strategy need to be up to date accordingly. Therefore, the energy management problem can be formulated as a Markov decision process (MDP) problem and solved by way of DDPG. PHET and the surrounding driving circumstances act as the environment, and the DDPG-based EMS functions as the agent. The basic components of DRL, i.e., state, action, and reward, need to be well-designed to solve the energy management problem for PHET. In this paper, we consider the energy consumption of the whole system and the balance of energy consumption and battery degradation. Hence, the state variables comprise vehicle speed, acceleration, SoC, and SoH. The whole state spaceS is shown below:
(23)S=V,acc,SoC,SoH,where V and acc are the vehicle speed and acceleration, respectively; these variables are vital parameters characterizing the vehicle operating state.Engine power was chosen as the action in this paper. Therefore, the action space can be defined as(24)action=PengPeng∈0,172kw.At the same time, EMS is aimed at decreasing fuel consumption, retaining the battery SoC in the adequate range, and reducing the cost of battery degradation, so the reward can be defined as:(25)J=αfuelt+elect+βSoCt−SoCref2+γSoHt−SoHref,where J is the objective function defined in energy management, α is the weight of fuel consumption, β is the weight of battery power maintenance, γ is the weight of battery degradation cost, fuel is the fuel consumption, and elec is the electric energy consumption. SoCref is the reference value of battery SoC, which is selected as 0.3 to prevent the damage to the battery from overdischarge. SoHref is the reference value of battery health status, which is selected as 1.0. Meanwhile, each powertrain component must meet the required control range in the control process.
(26)SoCmin≤SoC≤SoCmax,Txmin≤Tx≤Txmax,wxmin≤wx≤wxmax,x=eng,MG1,MG2,Imin≤Ibat≤Imax.As the structural characteristics of the PHET can be known, the engine speed and vehicle speed are decoupled from each other, considering that the engine efficiency fluctuates widely with the power variation. So to obtain better fuel economy, the control strategy and the work balance of the motor can be used to make the engine constantly work in the high-efficiency interval. The detailed hyperparameters of the DDPG are given in Table6, and the procedure of DDPG algorithm is shown in Algorithm 1.Table 6
Parameters used for DDPG.
Description (unit)ValueNumber of hidden layer of actor and critic (target) network3Neurons distribution of the actor and critic (target) network200, 100, 50Initial learning rate of actor-network0.001Initial learning rate of critic-network0.002Experience pool size50000Experience replay methodPrioritized experience replayMinibatch size64Discount factor0.9Soft update factor0.01Algorithm 1: Procedures of DDPG algorithm.
DDPG algorithmRandomly initialize critic-networkQs,aθμ and actor μsθμ with weights θμ and θμ, initialize target network Q′ and μ′ with weight θQ′⟵θQ, θμ′⟵θμ, initialize replay buffer Rforepisode=1: M dofort=1, T doSelect actionat=μstθμ according to the current policy and exploration noiseExecute actionat and observe reward rt and observe new state st+1Store transitionst,at,rt,st+1 in RSample a random minibatch ofN transitions st,at,rt,st+1 from RSetyi=ri+γQ′si+1,μ′si+1θQ′Update critic by minimizing the loss:L=1/N∑iyi−Qsi,aiθQ2Update the actor policy using the sampled policy gradient:∇θμJ≈1/N∑i∇aQs,aθQs=si,a=μsi∇θμμsθμsiUpdate the target networks:θQ′⟵τθQ+1−τθQ′θμ′⟵τθμ+1−τθμ′end forend forAs shown in Figure8, the optimal brake-specific fuel consumption (BSFC) curve is constructed in the engine map. Within the operation, any engine power corresponds to a speed-torque pair on the curve. While ensuring fuel economy, the action space dimension can be efficiently decreased so that the agent can find the optimum solution in a smaller action space, which speeds up the convergence of reinforcement learning [40].Figure 8
Engine operating curve.


### 4.3. Transfer Deep Reinforcement Learning

TL is a distinctive machine learning approach that can be used to solve problems in different but relevant domains, avoiding the need to retrain the network once the domain name is changed, which is quite expensive and inconvenient.When two domains are alike, the network parameters can be retained in the new domain for reuse by the transfer learning method. Given a source domainMs and a target domain Mt, the objective of transfer learning is to learn the optimum policy π∗ of the target domain Mt from the source domain Ms. The source domain Ms provides the prior knowledge Ds accessible to the target domain Mt. Therefore, by using the information from Ds, the agent can learn faster and better in the target domain Mt. Since the driving cycles of the source and target domains are interrelated and have the same feature space, the source domain knowledge can be transferred to the target domain.Dependent on the transferability, the basic DTL algorithm is combined with the DDPG algorithm, to transfer the knowledge of EMSs between source driving cycles and target usage scenarios.


### 4.4. Overall Control Framework

The overall control framework of this paper is shown in Figure9. The K-means and SVM hybrid algorithm are applied for feature extraction and classifying driving segments. The representative driving cycles of PHET under three common usage scenarios are constructed based on the Markov and Monte Carlo simulation method. The representative driving cycles can fully reflect the characteristics of natural usage scenarios; then, the DRL is used to train under the representative driving cycles thoroughly. Combining the driving pattern recognition results and DTL technique, the DDPG agent transfers the pretrained network from the corresponding source domain to real driving cycles. Also, fuel consumption, power consumption, battery aging, and SoC maintenance are used as reward functions to reduce the total driving cost. The simulation results are analyzed in the next section to evaluate the EMS proposed in this paper.Figure 9
Overall control framework.


## 4.1. The Formulation of DDPG Algorithm

RL is a machine learning technique that learns from the interplay between agent and environment, which derives the optimal control policy by trial-and-error methods in performing the behavior and getting feedback from the environment. In the process of reinforcement learning, the agent updates and learns the strategy by obtaining the goal of maximum cumulative reward, and the accumulative discounted expected reward is(19)Rt=∑i=tTγi−trsi,ai,where t is the time step, γ is a discount factor of size 0 to 1 to adjust the importance between immediate and future rewards, and r is the immediate reward. The optimum control strategy learned by reinforcement learning gives the mapping from state to action. Obtaining the maximum cumulative reward is only the means, and finding the method to solve the optimal control problem is the ultimate goal.Recent years have witnessed a spurt of machine learning progress, while engineering control problems have become increasingly complex. DDPG was created to manage problems in multidimensional continuous state and action spaces [39]. DDPG is an off-policy, off-model DRL, a deep reinforcement learning algorithm developed based on an actor-critic architecture. Actor-network μsθμ is input with state observations, which are then mapped to a deterministic action by the neural network. Critical-network Qsθμ is input with observations of the action taken by the actor-network and the current state to evaluate the merit of the present action.In addition, the experience replay buffer is introduced into the DDPG to reduce the randomness and dependence among samples, which can also reduce the bias arising from the estimation of the value function. However, to minimize the bias arising from taking a single critic forQ-value estimation and a single actor-network for action selection, target actor-network μ′sθμ′ and target critic-network Q′sθQ′ are introduced to estimate the Q-value:
(20)yt=rt+γQ′st+1,μ′st+1θμ′θQ′,where L is the average loss and N is the fixed size of the minibatch, arbitrarily chosen from the empirical replay buffer. The selection action by the actor-network μsθμ is to maximize the Q-value. Thus, the parameter update θμ can be solved numerically by applying the gradient method. The derived chain rule is applied as follows:
(21)∇θμJ≈1N∑i∇aQsi,aiθQ∇θμμsiθμ.Meanwhile, the target networksμ′ and Q′ using time lag updates can significantly improve the stability of learning:
(22)θQ′⟵τθQ+1−τθQ′,θμ′⟵τθμ+1−τθμ′,where τ is the soft update factor and θ and θ′ are the original network and target network parameters, respectively. The DDPG algorithm also improves the training efficiency by using priority experience replay to enhance the training process’s stability and the model’s robustness, while extra noise is introduced to the output of the actor-network to allow the DDPG agent to perform better exploration and select the correct action. In this paper, softmax action noise (SAN) noise is employed.


## 4.2. Energy Management Problem

The core opinion of EMS is to transform the energy management problem into an optimal control problem with distinct optimization objectives and control restrictions. EMS needs to make a reasonable allocation of power under the premise of satisfying vehicle demand power. After each power allocation decision, the driving environment, vehicle state switch, and strategy need to be up to date accordingly. Therefore, the energy management problem can be formulated as a Markov decision process (MDP) problem and solved by way of DDPG. PHET and the surrounding driving circumstances act as the environment, and the DDPG-based EMS functions as the agent. The basic components of DRL, i.e., state, action, and reward, need to be well-designed to solve the energy management problem for PHET. In this paper, we consider the energy consumption of the whole system and the balance of energy consumption and battery degradation. Hence, the state variables comprise vehicle speed, acceleration, SoC, and SoH. The whole state spaceS is shown below:
(23)S=V,acc,SoC,SoH,where V and acc are the vehicle speed and acceleration, respectively; these variables are vital parameters characterizing the vehicle operating state.Engine power was chosen as the action in this paper. Therefore, the action space can be defined as(24)action=PengPeng∈0,172kw.At the same time, EMS is aimed at decreasing fuel consumption, retaining the battery SoC in the adequate range, and reducing the cost of battery degradation, so the reward can be defined as:(25)J=αfuelt+elect+βSoCt−SoCref2+γSoHt−SoHref,where J is the objective function defined in energy management, α is the weight of fuel consumption, β is the weight of battery power maintenance, γ is the weight of battery degradation cost, fuel is the fuel consumption, and elec is the electric energy consumption. SoCref is the reference value of battery SoC, which is selected as 0.3 to prevent the damage to the battery from overdischarge. SoHref is the reference value of battery health status, which is selected as 1.0. Meanwhile, each powertrain component must meet the required control range in the control process.
(26)SoCmin≤SoC≤SoCmax,Txmin≤Tx≤Txmax,wxmin≤wx≤wxmax,x=eng,MG1,MG2,Imin≤Ibat≤Imax.As the structural characteristics of the PHET can be known, the engine speed and vehicle speed are decoupled from each other, considering that the engine efficiency fluctuates widely with the power variation. So to obtain better fuel economy, the control strategy and the work balance of the motor can be used to make the engine constantly work in the high-efficiency interval. The detailed hyperparameters of the DDPG are given in Table6, and the procedure of DDPG algorithm is shown in Algorithm 1.Table 6
Parameters used for DDPG.
Description (unit)ValueNumber of hidden layer of actor and critic (target) network3Neurons distribution of the actor and critic (target) network200, 100, 50Initial learning rate of actor-network0.001Initial learning rate of critic-network0.002Experience pool size50000Experience replay methodPrioritized experience replayMinibatch size64Discount factor0.9Soft update factor0.01Algorithm 1: Procedures of DDPG algorithm.
DDPG algorithmRandomly initialize critic-networkQs,aθμ and actor μsθμ with weights θμ and θμ, initialize target network Q′ and μ′ with weight θQ′⟵θQ, θμ′⟵θμ, initialize replay buffer Rforepisode=1: M dofort=1, T doSelect actionat=μstθμ according to the current policy and exploration noiseExecute actionat and observe reward rt and observe new state st+1Store transitionst,at,rt,st+1 in RSample a random minibatch ofN transitions st,at,rt,st+1 from RSetyi=ri+γQ′si+1,μ′si+1θQ′Update critic by minimizing the loss:L=1/N∑iyi−Qsi,aiθQ2Update the actor policy using the sampled policy gradient:∇θμJ≈1/N∑i∇aQs,aθQs=si,a=μsi∇θμμsθμsiUpdate the target networks:θQ′⟵τθQ+1−τθQ′θμ′⟵τθμ+1−τθμ′end forend forAs shown in Figure8, the optimal brake-specific fuel consumption (BSFC) curve is constructed in the engine map. Within the operation, any engine power corresponds to a speed-torque pair on the curve. While ensuring fuel economy, the action space dimension can be efficiently decreased so that the agent can find the optimum solution in a smaller action space, which speeds up the convergence of reinforcement learning [40].Figure 8
Engine operating curve.


## 4.3. Transfer Deep Reinforcement Learning

TL is a distinctive machine learning approach that can be used to solve problems in different but relevant domains, avoiding the need to retrain the network once the domain name is changed, which is quite expensive and inconvenient.When two domains are alike, the network parameters can be retained in the new domain for reuse by the transfer learning method. Given a source domainMs and a target domain Mt, the objective of transfer learning is to learn the optimum policy π∗ of the target domain Mt from the source domain Ms. The source domain Ms provides the prior knowledge Ds accessible to the target domain Mt. Therefore, by using the information from Ds, the agent can learn faster and better in the target domain Mt. Since the driving cycles of the source and target domains are interrelated and have the same feature space, the source domain knowledge can be transferred to the target domain.Dependent on the transferability, the basic DTL algorithm is combined with the DDPG algorithm, to transfer the knowledge of EMSs between source driving cycles and target usage scenarios.


## 4.4. Overall Control Framework

The overall control framework of this paper is shown in Figure9. The K-means and SVM hybrid algorithm are applied for feature extraction and classifying driving segments. The representative driving cycles of PHET under three common usage scenarios are constructed based on the Markov and Monte Carlo simulation method. The representative driving cycles can fully reflect the characteristics of natural usage scenarios; then, the DRL is used to train under the representative driving cycles thoroughly. Combining the driving pattern recognition results and DTL technique, the DDPG agent transfers the pretrained network from the corresponding source domain to real driving cycles. Also, fuel consumption, power consumption, battery aging, and SoC maintenance are used as reward functions to reduce the total driving cost. The simulation results are analyzed in the next section to evaluate the EMS proposed in this paper.Figure 9
Overall control framework.


## 5. Results and Discussion

### 5.1. Precondition for Validation

During the training process, representative driving cycles under each usage scenario are used for training, while real-world driving data of PHETs under three usage scenarios are selected for application validation. Since the constructed representative driving cycles have limited duration and are shorter than the all-electric range, the previously established driving cycles are repeated four times to constitute the training sets, and the duration of each driving cycle is about 7200 s.In the training and application processes, the initial value of SoC is set as 0.6 to maintain a sufficient charge/discharge margin, and the ambient temperature is assumed to be 313 K. Meanwhile, during the application process, the usage scenario recognized by DPR will not change during the entire driving process. Meanwhile, in this paper, we assume that the implementation of all energy management strategies is based on an intelligent traffic system based on V2X technology and collects and uploads global positioning system (GPS) data of the target truck using a truck terminal recorder. The intelligent traffic system (ITS) can thus obtain specific traffic flow information of each driving road, and when the target truck performs its operation task, the cloud computing center calculates the traffic flow information of the truck’s planned operation route to characterize the average speed of the traffic flow. The average speed of traffic flow is equal to the average instantaneous speed of all vehicles on a certain road section. The calculation formula is as follows:(27)v¯=nl∑iti=n∑i1/vi,where n is the number of driving segments and l is the distance of each driving cycle. The real-world driving cycles under three usage scenarios are shown in Figure 10.Figure 10
Real driving cycle of different scenarios. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)


### 5.2. Driving Pattern Recognition Performance Verification

The pattern recognition results in each usage scenario are shown in Figure11. The usage scenario with the largest percentage of DPR recognition in the first 1000 s will be identified as the usage scenario in which this driving cycle belongs. In the process of rolling accumulation recognition, Figure 11(a) shows the real-world driving pattern recognition result in scenario A. The first 75 s have not been recognized due to the fact that characteristic parameters of scenario A and scenario B are similar. And in the start-up period, it is not sufficient to reflect the complete characteristics. But after 75 seconds, scenario A is well recognized. Figure 11(b) shows that the recognizer can accurately recognize scenario B. Figure 11(c) shows the real-world driving pattern recognition result in scenario C. The characteristics of scenario C are biased toward a higher average speed. Due to the lack of information in the start-up period for the driving cycle, the average speed and other characteristic parameters are not obvious enough, so the first 525 s is recognized as other scenarios, but the subsequent are recognized as ideal scenario. Within the first 1000 seconds, scenario C still accounts for the largest proportion. Therefore, the established DPR model can recognize the real-world usage scenario well, which verifies that this DPR model is valid and ensure the accurate determination of the subsequent transfer learning source domain.Figure 11
The driving pattern recognition results. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)


### 5.3. Validation of Transfer Learning

This section is aimed at evaluating the transferability across representative driving cycles and real-world driving cycles. The strategy proposed in this paper is compared with the DDPG-based EMS with SoH consideration but without TL techniques.Meanwhile, some performance metrics are explicitly characterized for the transfer task. According to the definition in [41], four metrics are chosen to measure the benefit of the transfer task: jumpstart, asymptotic performance, total reward, and time-to-threshold, as defined below.Jumpstart performance: the initial performance of the agent. The mean reward before training is used to evaluate itAsymptotic performance: the average reward when the agent reaches the convergence stage. The mean reward of the convergence interval is used to evaluate itTime to threshold: the iterations required to converge to the target value. The number of iterations is used to evaluate itTable7 shows the results of the TL evaluation metrics, and the measures are mainly based on the reward values, while the specific average reward values in the three usage scenarios are shown in Figure 12.Table 7
Performance evaluation of transfer.
Usage scenarioStrategyJumpstart performanceAsymptotic performanceTime to thresholdAProposed4.04582.085918Common DDPG11.28872.210840BProposed0.66670.973217Common DDPG3.45471.421039CProposed0.78902.287121Common DDPG9.06163.296541Figure 12
Mean reward trajectories in the two DRL techniques. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)The proposed strategy has a significant improvement in jumpstart performance, asymptotic performance, and time to threshold. Figure12 shows that the proposed strategy decreases the exploration time in the training process, which also demonstrates that the proposed strategy has more knowledge about the environment. Although no exploration was required, the average reward showed that the proposed strategy still maintained the same or higher level as the baseline.It is noted that the weighting factors in the reward take the same value in all usage scenarios. Furthermore, as can be seen from the comparison of Figure13 with Figure 12, the convergence speed of the DDPG algorithm without SoH consideration is faster than the DDPG algorithm with SoH consideration. From the algorithmic point of view, the increase in the number of states indicates an increase in the neurons of the neural network’s input layer and a more complicated network gradient optimization, which influences both convergence and exploration.Figure 13
Mean reward trajectories in the training process.


### 5.4. Validation of Overall Driving Cost Optimization and Degradation Control

To further evaluate the proposed strategy, this section is aimed at evaluating the driving cost of the proposed strategy for each usage scenario task and the performance on the lifetime extension of the lithium-ion battery (LIB).The proposed strategy is compared with the DDPG-based EMS without DPR technique, DDPG-based EMS without SoH consideration, and DP. It is worth noting that the generalization ability of the DDPG algorithm has been demonstrated in the previous literature [42, 43]. The corresponding observation can make a better strategy without knowing the environmental changes. However, in this case, the intelligence can still generally not make the corresponding optimal strategy.So the DDPG-based EMS without DPR technique is identified as the DDPG algorithm, which uses the China heavy-duty commercial vehicle test cycle (CHTC) as the training dataset and is directly applied for testing in real-world usage scenarios. CHTC is the standard driving cycle that can reflect the actual road traffic conditions of PHETs. The DDPG-based EMS without DPR also considers the battery degradation cost. The speed trajectory of CHTC is shown in Figure14.Figure 14
China heavy-duty commercial vehicle test cycle (CHTC).The objective of the EMS problem established is to minimize the cost function, which includes three components of fuel consumption, electricity consumption, and battery degradation cost. The fuel economy is directly reflected by fuel consumption. The fuel economy of different EMSs under different real-world usage scenarios is displayed in Table8. The results reveal that in all three usage scenarios, the EMS with SoH consideration tends to yield fuel consumption increases due to the increase of the constraint on the battery degradation cost in the reward function. Reducing battery abuse and the corresponding fuel consumption will increase while fulfilling the vehicle demand power.Table 8
Comparative results of fuel economy.
Usage scenarioStrategyFuel consumptionPercentage fuel economyAProposed strategy3844.4 g91.83%DDPG without DPR4219.9 g83.66%DDPG without battery limit3670.2 g96.17%DP3530.3 g100%BProposed strategy2907.5 g93.81%DDPG without DPR3367.1 g81.00%DDPG without battery limit2830.5 g96.36%DP2727.6 g100%CProposed strategy7170.3 g90.10%DDPG without DPR7285.3 g88.77%DDPG without battery limit6693.5 g96.00%DP6467.3 g100%The results show that the DP algorithm has the lowest fuel consumption due to its global optimality, which is the benchmark of EMS but is difficult to apply practically [44]. The proposed strategy shows better performance in fuel economy than the DDPG-based EMS without the DPR technique, which achieves 91.83%, 93.81%, and 90.10% of DP in fuel economy on average in each usage scenario. However, compared with DDPG-based EMSs without SoH consideration, the proposed strategy increases only an average of about 5% fuel consumption.The engine output power distribution of the EMSs in the three usage scenarios is shown in Figure15. The distribution of these strategies is not precisely the same. The majority of the engine operating points of DP-based EMS is concentrated in zero- or low-power areas, and the percentage of operating points in the high power areas is less than that of the proposed strategy and DDPG-based EMS without DPR. It can also be seen that the proposed strategy’s distribution proportion is closer to that of the DP-based EMS, and the percentage of engine operating points in the high-power areas is lower in the proposed strategy than in the DDPG-based EMS without DPR.Figure 15
Distribution proportion of engine working points. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)Another essential part of the reward function of this paper is the battery SoC maintenance. The SoC of different EMSs are shown in Figure16, and all terminal values of SoC trajectories are effectively maintained in the range of 0.25~0.3.Figure 16
SOC of different EMSs. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)Also, the trend of SoC variation throughout the process reflects the problem that this paper will target: an optimum energy management strategy for a specialized driving cycle may become suboptimal for another driving cycle.For DDPG-based EMS without DPR technique, which uses CHTC as a training dataset and applies to each usage scenario, it appears suboptimal in terms of fuel consumption.The agent trained by CHTC has the generalization ability to choose the action by the most similar state value when facing a brand new driving cycle, and the action orientation is more representative of CHTC. As shown in Figure16, at time step 3000 in scenario A, the agent of CHTC chooses to charge the battery to compensate for how fast the electricity is consumed in the early stage, which also leads to an increase in fuel consumption. Meanwhile, throughout the three usage scenarios, CHTC’s agent prefers to use electricity drive while increasing the frequency of substantial battery charging and discharging. Of course, battery degradation is inevitable due to the gradual use of the battery. Still, the SoC trajectory of the proposed strategy has a slower decreasing trend and less fluctuation during the whole process. Meanwhile, the proposed strategy can adopt the power allocation strategy that reflects the characteristics of the real usage scenario.It is worth noting that the average speed of the CHTC is higher, and its characteristic parameters are closer to those of scenario C. When applied directly in the real scenario of scenario C, it can show better control effects but is inferior in other scenarios. From here, it can also be noticed that the application value of the proposed strategy makes it possible for the agent to get the optimal control effect faster and better in various usage scenarios.Also to compare the computational efficiency of the proposed strategy, Table9 depicts the computation time of the proposed algorithm compared to the commonly used energy management algorithm DP. From the comparison results, it can be seen that the proposed strategy has a shorter computation time compared to the DP algorithm, thus showing that the proposed strategy is more usable for practical application.Table 9
Consumption of solution time.
Usage scenarioStrategyTime (s)AProposed5.741DP344.47BProposed22.51DP442.62CProposed11.80DP299.45∗A i7-10810H CPU @2.2 GHz microprocessor with 16 GB RAM was used.Tables10 and 11 further compare the performance of degradation control and the total driving cost per 100 km. It is shown in Table 10 that the proposed strategy achieves the best degradation control in all usage scenarios and improves the degradation control performance by 48.46% in scenario A, 57.95% in scenario B, and 36.99% in scenario C when considering battery health. In the cost comparison, the proposed strategy is contrasted with DDPG-based EMS without battery degradation consideration and DQL-based EMS with battery degradation consideration. DQL algorithm is a DRL algorithm that handles segmented action states and has been shown to achieve good control results in the EMS of hybrid vehicles [15, 45, 46]. Under this premise, the cost of different strategies per 100 km under the three real-world usage scenarios is listed in Table 11. The battery degradation cost can be replaced by the battery replacement price of ￥970.63/kWh [47], and the diesel oil price is ￥7.5/kg. It can be concluded that although the common DDPG strategies show better fuel economy in these three usage scenarios, the neglected relatively high cost of battery degradation leads to higher overall costs than the proposed strategy. Besides, the proposed strategy leads to the best performance in reducing the total driving cost. The total cost of the proposed strategy has been reduced by 17.76% in scenario A, even 8.51% in scenario B, and 7.12% in scenario C compared with the DDPG-based EMS, which is the most advanced DRL-based EMS algorithm. As shown by the results, it can also be seen that the proposed strategy makes application to the continuous action state space and therefore has a lower driving cost compared to the DQL algorithm applied in [15] under segmented usage scenarios. By comparing all aspects, it is obvious that the proposed strategy in this paper is better than the existing RL strategy.Table 10
Comparison of the degradation control performance.
Usage scenarioStrategyFinal SOHCapacity lossPerformanceAProposed0.9990800.091966%100%DDPG without battery limit0.9982090.179053%51.36%BProposed0.9992860.07139%100%DDPG without battery limit0.9983020.1678%42.05%CProposed0.9989360.106394%100%DDPG without battery limit0.9983110.168851%63.01%Table 11
Comparative results of driving cost per 100 km.
Usage scenarioStrategyFuel consumption (CNY)Battery degradation cost (CNY)Overall driving cost (CNY)PerformanceAProposed strategy257.51086.609344.119100%DDPG without battery limit257.377156.020413.39783.24%DQL-based302.274220.303522.57765.85%BProposed strategy220.20456.953277.1577100%DDPG without battery limit218.33880.907299.244292.62%DQL-based271.968154.508426.475664.99%CProposed strategy175.38742.464217.464100%DDPG without battery limit166.75467.391234.14592.88%DQL-based185.15598.270283.42576.73%


## 5.1. Precondition for Validation

During the training process, representative driving cycles under each usage scenario are used for training, while real-world driving data of PHETs under three usage scenarios are selected for application validation. Since the constructed representative driving cycles have limited duration and are shorter than the all-electric range, the previously established driving cycles are repeated four times to constitute the training sets, and the duration of each driving cycle is about 7200 s.In the training and application processes, the initial value of SoC is set as 0.6 to maintain a sufficient charge/discharge margin, and the ambient temperature is assumed to be 313 K. Meanwhile, during the application process, the usage scenario recognized by DPR will not change during the entire driving process. Meanwhile, in this paper, we assume that the implementation of all energy management strategies is based on an intelligent traffic system based on V2X technology and collects and uploads global positioning system (GPS) data of the target truck using a truck terminal recorder. The intelligent traffic system (ITS) can thus obtain specific traffic flow information of each driving road, and when the target truck performs its operation task, the cloud computing center calculates the traffic flow information of the truck’s planned operation route to characterize the average speed of the traffic flow. The average speed of traffic flow is equal to the average instantaneous speed of all vehicles on a certain road section. The calculation formula is as follows:(27)v¯=nl∑iti=n∑i1/vi,where n is the number of driving segments and l is the distance of each driving cycle. The real-world driving cycles under three usage scenarios are shown in Figure 10.Figure 10
Real driving cycle of different scenarios. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)


## 5.2. Driving Pattern Recognition Performance Verification

The pattern recognition results in each usage scenario are shown in Figure11. The usage scenario with the largest percentage of DPR recognition in the first 1000 s will be identified as the usage scenario in which this driving cycle belongs. In the process of rolling accumulation recognition, Figure 11(a) shows the real-world driving pattern recognition result in scenario A. The first 75 s have not been recognized due to the fact that characteristic parameters of scenario A and scenario B are similar. And in the start-up period, it is not sufficient to reflect the complete characteristics. But after 75 seconds, scenario A is well recognized. Figure 11(b) shows that the recognizer can accurately recognize scenario B. Figure 11(c) shows the real-world driving pattern recognition result in scenario C. The characteristics of scenario C are biased toward a higher average speed. Due to the lack of information in the start-up period for the driving cycle, the average speed and other characteristic parameters are not obvious enough, so the first 525 s is recognized as other scenarios, but the subsequent are recognized as ideal scenario. Within the first 1000 seconds, scenario C still accounts for the largest proportion. Therefore, the established DPR model can recognize the real-world usage scenario well, which verifies that this DPR model is valid and ensure the accurate determination of the subsequent transfer learning source domain.Figure 11
The driving pattern recognition results. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)


## 5.3. Validation of Transfer Learning

This section is aimed at evaluating the transferability across representative driving cycles and real-world driving cycles. The strategy proposed in this paper is compared with the DDPG-based EMS with SoH consideration but without TL techniques.Meanwhile, some performance metrics are explicitly characterized for the transfer task. According to the definition in [41], four metrics are chosen to measure the benefit of the transfer task: jumpstart, asymptotic performance, total reward, and time-to-threshold, as defined below.Jumpstart performance: the initial performance of the agent. The mean reward before training is used to evaluate itAsymptotic performance: the average reward when the agent reaches the convergence stage. The mean reward of the convergence interval is used to evaluate itTime to threshold: the iterations required to converge to the target value. The number of iterations is used to evaluate itTable7 shows the results of the TL evaluation metrics, and the measures are mainly based on the reward values, while the specific average reward values in the three usage scenarios are shown in Figure 12.Table 7
Performance evaluation of transfer.
Usage scenarioStrategyJumpstart performanceAsymptotic performanceTime to thresholdAProposed4.04582.085918Common DDPG11.28872.210840BProposed0.66670.973217Common DDPG3.45471.421039CProposed0.78902.287121Common DDPG9.06163.296541Figure 12
Mean reward trajectories in the two DRL techniques. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)The proposed strategy has a significant improvement in jumpstart performance, asymptotic performance, and time to threshold. Figure12 shows that the proposed strategy decreases the exploration time in the training process, which also demonstrates that the proposed strategy has more knowledge about the environment. Although no exploration was required, the average reward showed that the proposed strategy still maintained the same or higher level as the baseline.It is noted that the weighting factors in the reward take the same value in all usage scenarios. Furthermore, as can be seen from the comparison of Figure13 with Figure 12, the convergence speed of the DDPG algorithm without SoH consideration is faster than the DDPG algorithm with SoH consideration. From the algorithmic point of view, the increase in the number of states indicates an increase in the neurons of the neural network’s input layer and a more complicated network gradient optimization, which influences both convergence and exploration.Figure 13
Mean reward trajectories in the training process.


## 5.4. Validation of Overall Driving Cost Optimization and Degradation Control

To further evaluate the proposed strategy, this section is aimed at evaluating the driving cost of the proposed strategy for each usage scenario task and the performance on the lifetime extension of the lithium-ion battery (LIB).The proposed strategy is compared with the DDPG-based EMS without DPR technique, DDPG-based EMS without SoH consideration, and DP. It is worth noting that the generalization ability of the DDPG algorithm has been demonstrated in the previous literature [42, 43]. The corresponding observation can make a better strategy without knowing the environmental changes. However, in this case, the intelligence can still generally not make the corresponding optimal strategy.So the DDPG-based EMS without DPR technique is identified as the DDPG algorithm, which uses the China heavy-duty commercial vehicle test cycle (CHTC) as the training dataset and is directly applied for testing in real-world usage scenarios. CHTC is the standard driving cycle that can reflect the actual road traffic conditions of PHETs. The DDPG-based EMS without DPR also considers the battery degradation cost. The speed trajectory of CHTC is shown in Figure14.Figure 14
China heavy-duty commercial vehicle test cycle (CHTC).The objective of the EMS problem established is to minimize the cost function, which includes three components of fuel consumption, electricity consumption, and battery degradation cost. The fuel economy is directly reflected by fuel consumption. The fuel economy of different EMSs under different real-world usage scenarios is displayed in Table8. The results reveal that in all three usage scenarios, the EMS with SoH consideration tends to yield fuel consumption increases due to the increase of the constraint on the battery degradation cost in the reward function. Reducing battery abuse and the corresponding fuel consumption will increase while fulfilling the vehicle demand power.Table 8
Comparative results of fuel economy.
Usage scenarioStrategyFuel consumptionPercentage fuel economyAProposed strategy3844.4 g91.83%DDPG without DPR4219.9 g83.66%DDPG without battery limit3670.2 g96.17%DP3530.3 g100%BProposed strategy2907.5 g93.81%DDPG without DPR3367.1 g81.00%DDPG without battery limit2830.5 g96.36%DP2727.6 g100%CProposed strategy7170.3 g90.10%DDPG without DPR7285.3 g88.77%DDPG without battery limit6693.5 g96.00%DP6467.3 g100%The results show that the DP algorithm has the lowest fuel consumption due to its global optimality, which is the benchmark of EMS but is difficult to apply practically [44]. The proposed strategy shows better performance in fuel economy than the DDPG-based EMS without the DPR technique, which achieves 91.83%, 93.81%, and 90.10% of DP in fuel economy on average in each usage scenario. However, compared with DDPG-based EMSs without SoH consideration, the proposed strategy increases only an average of about 5% fuel consumption.The engine output power distribution of the EMSs in the three usage scenarios is shown in Figure15. The distribution of these strategies is not precisely the same. The majority of the engine operating points of DP-based EMS is concentrated in zero- or low-power areas, and the percentage of operating points in the high power areas is less than that of the proposed strategy and DDPG-based EMS without DPR. It can also be seen that the proposed strategy’s distribution proportion is closer to that of the DP-based EMS, and the percentage of engine operating points in the high-power areas is lower in the proposed strategy than in the DDPG-based EMS without DPR.Figure 15
Distribution proportion of engine working points. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)Another essential part of the reward function of this paper is the battery SoC maintenance. The SoC of different EMSs are shown in Figure16, and all terminal values of SoC trajectories are effectively maintained in the range of 0.25~0.3.Figure 16
SOC of different EMSs. (a) Scenario A. (b) Scenario B. (c) Scenario C.
(a)(b)(c)Also, the trend of SoC variation throughout the process reflects the problem that this paper will target: an optimum energy management strategy for a specialized driving cycle may become suboptimal for another driving cycle.For DDPG-based EMS without DPR technique, which uses CHTC as a training dataset and applies to each usage scenario, it appears suboptimal in terms of fuel consumption.The agent trained by CHTC has the generalization ability to choose the action by the most similar state value when facing a brand new driving cycle, and the action orientation is more representative of CHTC. As shown in Figure16, at time step 3000 in scenario A, the agent of CHTC chooses to charge the battery to compensate for how fast the electricity is consumed in the early stage, which also leads to an increase in fuel consumption. Meanwhile, throughout the three usage scenarios, CHTC’s agent prefers to use electricity drive while increasing the frequency of substantial battery charging and discharging. Of course, battery degradation is inevitable due to the gradual use of the battery. Still, the SoC trajectory of the proposed strategy has a slower decreasing trend and less fluctuation during the whole process. Meanwhile, the proposed strategy can adopt the power allocation strategy that reflects the characteristics of the real usage scenario.It is worth noting that the average speed of the CHTC is higher, and its characteristic parameters are closer to those of scenario C. When applied directly in the real scenario of scenario C, it can show better control effects but is inferior in other scenarios. From here, it can also be noticed that the application value of the proposed strategy makes it possible for the agent to get the optimal control effect faster and better in various usage scenarios.Also to compare the computational efficiency of the proposed strategy, Table9 depicts the computation time of the proposed algorithm compared to the commonly used energy management algorithm DP. From the comparison results, it can be seen that the proposed strategy has a shorter computation time compared to the DP algorithm, thus showing that the proposed strategy is more usable for practical application.Table 9
Consumption of solution time.
Usage scenarioStrategyTime (s)AProposed5.741DP344.47BProposed22.51DP442.62CProposed11.80DP299.45∗A i7-10810H CPU @2.2 GHz microprocessor with 16 GB RAM was used.Tables10 and 11 further compare the performance of degradation control and the total driving cost per 100 km. It is shown in Table 10 that the proposed strategy achieves the best degradation control in all usage scenarios and improves the degradation control performance by 48.46% in scenario A, 57.95% in scenario B, and 36.99% in scenario C when considering battery health. In the cost comparison, the proposed strategy is contrasted with DDPG-based EMS without battery degradation consideration and DQL-based EMS with battery degradation consideration. DQL algorithm is a DRL algorithm that handles segmented action states and has been shown to achieve good control results in the EMS of hybrid vehicles [15, 45, 46]. Under this premise, the cost of different strategies per 100 km under the three real-world usage scenarios is listed in Table 11. The battery degradation cost can be replaced by the battery replacement price of ￥970.63/kWh [47], and the diesel oil price is ￥7.5/kg. It can be concluded that although the common DDPG strategies show better fuel economy in these three usage scenarios, the neglected relatively high cost of battery degradation leads to higher overall costs than the proposed strategy. Besides, the proposed strategy leads to the best performance in reducing the total driving cost. The total cost of the proposed strategy has been reduced by 17.76% in scenario A, even 8.51% in scenario B, and 7.12% in scenario C compared with the DDPG-based EMS, which is the most advanced DRL-based EMS algorithm. As shown by the results, it can also be seen that the proposed strategy makes application to the continuous action state space and therefore has a lower driving cost compared to the DQL algorithm applied in [15] under segmented usage scenarios. By comparing all aspects, it is obvious that the proposed strategy in this paper is better than the existing RL strategy.Table 10
Comparison of the degradation control performance.
Usage scenarioStrategyFinal SOHCapacity lossPerformanceAProposed0.9990800.091966%100%DDPG without battery limit0.9982090.179053%51.36%BProposed0.9992860.07139%100%DDPG without battery limit0.9983020.1678%42.05%CProposed0.9989360.106394%100%DDPG without battery limit0.9983110.168851%63.01%Table 11
Comparative results of driving cost per 100 km.
Usage scenarioStrategyFuel consumption (CNY)Battery degradation cost (CNY)Overall driving cost (CNY)PerformanceAProposed strategy257.51086.609344.119100%DDPG without battery limit257.377156.020413.39783.24%DQL-based302.274220.303522.57765.85%BProposed strategy220.20456.953277.1577100%DDPG without battery limit218.33880.907299.244292.62%DQL-based271.968154.508426.475664.99%CProposed strategy175.38742.464217.464100%DDPG without battery limit166.75467.391234.14592.88%DQL-based185.15598.270283.42576.73%


## 6. Conclusion

This paper proposes a unique DTRL-based energy management with the assistance of DPR for PHET. The strategy proposed solves the slow convergence of control strategies based on DRL and improves the generalization ability of EMS for PHETs in each usage scenario. Meanwhile, the battery degradation cost is incorporated into the cost function, which greatly extends the battery life while ensuring the fuel consumption. The main results are drawn as follows:(1)
TheK-means and SVM hybrid algorithm are applied for feature extraction and classifying the driving condition segments. Three constructed representative driving cycles can truly reflect different usage scenarios using the Markov and Monte Carlo simulation method(2)
A driving pattern recognition approach using the LVQ neural network is designed to determine the transfer source domain. In addition, the DDPG algorithm is combined with the DTL algorithm to accelerate the training convergence speed of knowledge transfer between EMSs, which improves the convergence speed by nearly 50%(3)
The proposed strategy also introduces the battery degradation cost into the optimization object along with the fuel consumption term. Compared with the existing state-of-the-art strategies, the proposed strategy shows better performances in terms of convergence speed, battery degradation attenuation, fuel economy, and driving cost. The results show that the proposed strategy can achieve the fastest convergence speed while extending battery life with minimal driving cost. The proposed strategy reduces the battery degradation cost by 48.46%, 57.95%, and 36.99% and the total drive cost by 17.76%, 8.51%, and 7.12% under segmented usage scenarios compared to the existing state-of-the-art strategy, respectivelyThe effectiveness of the control strategy is validated on a single PHET, and in the future, information from intelligent transportation system will be further integrated into the study of EMSs and the overall coordinated control of PHET fleet by ITS information.



---

*Source: 1875380-2023-02-13.xml*