# Reinforcement learning-based energy management for hybrid electric vehicles: A comprehensive up-to-date review on methods, challenges, and research gaps

Mohamed Nadir Boukoberine, Muhammad Fahad Zia, Tarek Berghout, Mohamed Benbouzid

► To cite this version:

Mohamed Nadir Boukoberine, Muhammad Fahad Zia, Tarek Berghout, Mohamed Benbouzid. Reinforcement learning-based energy management for hybrid electric vehicles: A comprehensive up-to-date review on methods, challenges, and research gaps. Energy and AI, 2025, 21, pp.100514. 10.1016/j.egyai.2025.100514. hal-05333739

HAL Id: hal-05333739

https://hal.science/hal-05333739v1

Submitted on 27 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Review

# Reinforcement learning-based energy management for hybrid electric vehicles: A comprehensive up-to-date review on methods, challenges, and research gaps

<figure><img src="image_2.png" /></figure>

Mohamed Nadir Boukoberine a, b, Muhammad Fahad Zia c, Tarek Berghout d, Mohamed Benbouzid b, e, d, *

## HIGHLIGHTS

* Comprehensive understanding of energy management for hybrid electric vehicles.

* Comprehensive understanding of reinforcement learning principles.

* Analysis of reinforcement learning methods for optimizing HEVs energy management.

* Key research directions and challenges in RL-based energy management for HEVs.

## GRAPHICAL ABSTRACT

<figure><img src="image_3.png" /></figure>

## ARTICLE INFO

Keywords:

Deep reinforcement learning

Energy management

## ABSTRACT

Reinforcement learning is widely used for control applications and has also been successfully implemented for efficient energy management within hybrid electric vehicles. Reinforcement learning algorithms offer various advantages, including fast convergence, broad applicability, stability, and robustness, particularly with the

<footnote>a Ecole Militaire Polytechnique, UER ELT, 16111 Alger, Algeria</footnote>

<footnote>b University of Brest, UMR CNRS 60 IRDL, 29238 Brest, France</footnote>

<footnote>c American University in Dubai, Department of Electrical and Computer Engineering, 28282 Dubai, United Arab Emirates</footnote>

<footnote>d University of Batna 2, Laboratory of Automation and Manufacturing Engineering, 05000 Batna, Algeria</footnote>

<footnote>e Logistics Engineering College, Shanghai Maritime University, Shanghai 201306, China</footnote>

<footnote>* Corresponding author at: University of Brest, UMR CNRS 60 IRDL, 29238 Brest, France.
E-mail address: mohamed.benbouzid@univ-brest.fr (M. Benbouzid).</footnote>
Electric vehicle

Reinforcement learning

Transfer learning

integration of deep and transfer learning. This paper provides a comprehensive understanding of reinforcement learning principles and a critical review of various reinforcement learning methods, states, actions, and rewards used to optimize the energy management performance of hybrid electric vehicles. Furthermore, the advantages and limitations of these algorithms are also discussed. This review reveals that deep reinforcement learning techniques show superior performance in handling complex energy management tasks thanks to their ability to learn from high-dimensional state spaces. Nevertheless, their implementation faces notable obstacles, including computational complexity and generalization across diverse driving conditions. Finally, key research directions for future work and challenges are highlighted in the domain of reinforcement-learning-based hybrid electric vehicle energy management.

# 1. Introduction

Global sales of electric vehicles (EVs) and hybrid electric vehicles (HEVs) have increased in recent years due to advancements in automation and mitigation of global warming [1]. According to the International Energy Agency (IEA), global sales of EVs increased to 6.6 million in 2021 from the previous year. By 2025, the global market for hybrid vehicles is expected to reach a valuation of roughly US$312.8 billion, with forecasts suggesting it could grow to about US$580.9 billion by 2034 [2]. Furthermore, plug-in hybrid electric vehicles (PHEVs) are projected to represent 9.23% of the Chinese car market in 2030 [3]. This rapid growth reflects the accelerating shift towards electric mobility on a global scale. HEVs typically combine an electric motor (EM) and an internal combustion engine (ICE) and are usually fed by two or more energy sources, such as batteries, fuel cells, supercapacitors, and gasoline, depending on the electrification degree. The hybridization of diverse energy sources has provided HEVs more flexibility in supplying energy than conventional vehicles. Furthermore, hybridization allows the combined utilization of their advantages by combining their unique generation and storage characteristics. This hybridization also mitigates the limitations of each source, resulting in enhanced performance of the hybrid power supply system (HPSS). The scheduling of these hybrid sources requires effectively implementing an energy management strategy (EMS) to control power sharing between sources while satisfying their constraints, conserving their lifetime, and increasing the overall system efficiency.

## 1.1. Energy management for HEVs

The main challenge in HEV is the limited availability of onboard energy. Therefore, the objective of an EMS is to provide an optimal energy supply to HEVs while reducing fuel consumption and emissions. However, the challenge lies in simultaneously satisfying the constraints and power demand, which often requires compromises to arrive at an optimal solution [4]. This is further complicated by the need to maintain vehicle performance at an optimal level [5]. Irrespective of the HEV type, the fuel economy and exhaust emission performance are heavily affected by the EMS, which has attracted considerable attention in recent research studies [6,7].

A general representation of the EMS is illustrated in Fig. 1 for HEV applications. Energy management strategies for HEVS can generally be classified into two categories [7]: rule-based and optimization-based strategies. A rule-based EMS is a simple and commonly known strategy that involves a set of predefined threshold-based rules on the control variables. The power system is supplied in different operating modes by mapping control variables to related thresholds, such as the frequency separation rule-based strategy [8] and rule-based fuzzy logic strategy [9]. Optimization-based approaches are typically realized by designing an objective function with a dynamic state-space model and specified constraints. These optimization-based EMSs can be solved by linear programming (LP), mixed-number linear programming (MILP), quadratic programming (QP), or nonlinear programming (NLP) solvers. Various optimization-based algorithms have been investigated in the literature, including Pontryagin's minimum principle (PMP) [10], equivalent consumption minimization strategy (ECMS) [11], model

prediction control (MPC) [12], and meta-heuristic methods [13,14]. The authors of [7] proposed a critical evaluation for numerous studies investigating EMSs dedicated to electric vehicles.

Energy management strategies have been widely used due to their effectiveness in real-time power allocation. Nevertheless, some rely mainly on expert knowledge, such as rule-based methods. PMP and MPC have a heavy computational burden. Accurate models are also required, and they are still sensitive to uncertainties in the power load profile or HPSS model [15]. These stochastic disruptions would lead to performance degradation of the HEV HPSS. Furthermore, the random characteristics of energy usage and flexible use of active loads lead to more uncertainty and complexity in HPSS power management. Hence, the main problem is the difficulty of developing an efficient general mathematical model that can fit most situations. Meta-heuristic methods such as the genetic algorithm (GA), particle swarm optimization (PSO), and dynamic programming (DP) can achieve global optimality. However, they cannot be implemented online because they require prior knowledge of the driving mission.

## 1.2. Motivation and necessity for an up-to-date review

To overcome the aforementioned limitations and achieve high adaptability power control for HEVs HPSS, the scientific research community has shown a strong interest in learning-based approaches exploiting artificial intelligence advances. As a core area of machine learning, reinforcement learning (RL) has the advantages of interactive trial and error-based self-learning in a dynamic environment [16]. It has been successfully implemented to address various real-world decision-making tasks under uncertain conditions. RL algorithms learn how to make control decisions by evaluating actions and obtaining rewards as feedback metrics derived from their own experience. The final goal is to learn the optimal policy with respect to a predefined objective function and constraints. The RL problem is typically modeled as a Markov Decision Process (MDP) [17]. Moreover, the system dynamics model is unnecessary, and the corresponding sensitivity can be omitted. Therefore, RL is considered to be a powerful tool for achieving near-optimal control decisions for nonlinear and uncertain systems. This can effectively improve the fuel economy, flexibility, and reliability of the HPSS in different vehicle types and for various missions. The integration of neural networks into RL approaches has also considerably boosted their performance by introducing a new branch known as deep reinforcement learning (DRL) [18,19].

Indeed, several studies have been carried out in recent years to investigate RL-based EMSs for HEVs. However, only a few reviews have been published in this field. The authors in [20] documented a comprehensive review of current advances in RL-based EMSs, discussing hybrid RL approaches that include mission information, predicted operating conditions, and algorithms such as MPC. In [21], the authors reviewed RL-based EMSs for fuel cell HEV (FCHEV), focusing on RL environments and the training process. The RL concept was well presented, and the RL impact on HEV energy management was demonstrated; however, RL algorithms were not described to enable the reader to gain a deep understanding of the RL technique application. In [22], only a few RL and DRL methods are discussed for HEV EMS. Reward objectives, states, actions, and challenges with RL methods are not discussed. RL and DRL-based EMS methods are surveyed in [23]
<figure><img src="image_4.png" /><figcaption>Fig. 1. General schematic representation of EMS for HEV.</figcaption></figure>

for electrified powertrains with their limitations. However, only a few DRL methods like deep deterministic policy gradient (DDPG) and deep Q-learning (DQL) are discussed. States and actions are not discussed. A review of DRL-based HEV EMS is provided in [24]. States, actions, and rewards of HEV EMS are also highlighted. The main challenges of real-world applications of DRL-based EMS are also discussed. However, DRL with transfer learning-based HEV EMS is not discussed. Offline and online RL methods are also not highlighted. Table 1 summarizes the main contributions of the existing review studies related to RL-based EMSs. This paper provides an in-depth exploration of both classical and deep RL (DRL) approaches, including their offline, online, and transfer learning-based implementations. Additionally, this review goes beyond algorithmic discussions to offer insights into practical challenges, real-world applications, and future research directions for RL-based HEV EMSs. The main contributions of this review are as follows:

* Detailed description of the HEV energy management problem, emphasizing key optimization objectives, parameters, and constraints.

* Comprehensive and structured overview of RL and DRL principles and DRL techniques for HEV EMSs is presented, including the analysis of state, action, and reward design, and comparative assessment of different algorithms to guide method selection.

* A detailed literature review is conducted, covering a wide range of RL-based EMSs applied in HEVs.

* RL potential on HEV HPSS energy management is discussed in terms of its efficacy, energy efficiency, adaptability, generalization, and training performance.

* Exploration of imitation learning as an alternative approach for efficient policy learning in HEV EMSs.

* Potential limitations and issues of the RL algorithm are identified for HEV energy management and emphasized as potential prospects for future exploration.

<table><caption>Table 1
Comparison with existing reviews related to RL-based HEV EMSs.</caption><thead><tr><th>Ref.</th><th>Main contributions</th></tr></thead><tbody><tr><td>[20]</td><td>Discussed simplex and hybrid RL techniques (only DQL and DDPG for DRL) which involve mission information, predicted operating conditions, and also algorithms like MPC.</td></tr><tr><td>[21]</td><td>A summary of the evolution of RL-based EMSs and a few DRL-based EMSs (DQL and DDPG) focusing on FCHEVs only and on RL environment construction and evolution process.</td></tr><tr><td>[22]</td><td>Simplex and hybrid RL for HEV EMS. In DRL-based EMS, only DQL and DDPG are provided.</td></tr><tr><td>[23]</td><td>A survey of RL and DRL (only DQL and DDPG) methods for EMS of electrified powertrains. Limitations of RL-based EMS methods: long learning time.</td></tr><tr><td>[24]</td><td>A summary of only DRL algorithms for HEV EMSs. States, actions, and rewards of DRL-based HEV EMSs. Main challenges and potential solutions for real-world applications.</td></tr><tr><td>[25]</td><td>A state-of-the-art survey on RL and DRL-based energy management strategies for hybrid power systems in EVs. Discussed algorithmic, powertrain, and environmental aspects of HEV EMSs.</td></tr><tr><td>Proposed</td><td>A comprehensive review of offline and online RL, DRL, and DRL with transfer learning-based HEV EMSs. Extensive implementation analysis of RL-based HEV EMS. Configuration analysis of mostly used RL and DRL techniques for HEV EMS. States, actions, rewards, and EV-type-based EMS analysis. Exploration of imitation learning with RL/DRL methods for HEV EMS. Discussion of challenges and perspectives for real-world applications of RL-based HEV EMSs: learning period, convergence, high precision training simulator, data quality, vehicle-to-everything (V2X) communication, reward function, and testing and evaluation.</td></tr></tbody></table>

## 1.3. Remainder work organization

The remainder of this paper is organized as follows: Section 2 introduces the HEV energy management problem. Section 3 presents an overview of the reinforcement learning concept. Section 4 critically explains RL algorithms and RL-based EMS for HEVs. Section 5 discusses
the critical evaluation of DRL algorithms and DRL-based EMS in the HEV domain. Section 6 highlights research gaps and future research directions.

## 2. Energy management strategies for HEVs

The energy management challenge in HEVs consists of optimizing power distribution among multiple energy sources—such as ICE, electric motor, battery, and in some cases, supercapacitors or waste heat recovery systems (WHRS) like the Organic Rankine Cycle (ORC). The goal is to meet objectives like minimizing fuel consumption, reducing emissions, and preserving battery longevity, all while maintaining optimal vehicle performance and drivability. This problem is inherently complex due to the dynamic driving environment, the interdependence of energy sources, and the necessity to comply with multiple operational constraints. The following sections provide an overview of the energy management problem in HEVs, outlining key optimization parameters and constraints that must be addressed.

### 2.1. Objectives of HEV energy management strategies

Objectives must be carefully balanced to address important EV goals such as fuel economy, emissions reduction, battery health, vehicle performance, and overall energy efficiency.

#### 2.1.1. Minimizing fuel consumption

One of the primary goals of HEV energy management is to reduce fuel consumption by optimizing the power split between the ICE and the electric motor. This is achieved by:

- Ensuring that the ICE operates in its most efficient regions (e.g., near peak efficiency points in the fuel consumption map).

- Prioritizing electric propulsion during low-power demand scenarios.

- Utilizing regenerative braking to recover energy that would otherwise be lost.

The significance of fuel consumption minimization lies in:

- Reduces operating costs by lowering fuel expenses.

- Decreases dependence on fossil fuels, promoting energy sustainability.

- Enhances driving range, particularly in plug-in HEVs (PHEVs).

Several obstacles related to minimizing fuel consumption can be addressed, including:

- Balancing fuel economy with performance demands (e.g., acceleration requirements).

- Handling varying driving conditions, which affect the optimal power split strategy.

- Managing ICE start-stop transitions to avoid excessive fuel consumption during frequent restarts.

#### 2.1.2. Reducing emissions

Energy management strategies must aim to minimize greenhouse gas emissions (CO₂) and pollutants (NOₓ, particulate matter, CO) by:

- Optimizing ICE operating conditions to stay within low-emission zones.

- Maximizing the use of the EMin urban/low-speed conditions.

- Coordinating battery charging strategies to reduce emissions from external power generation in PHEVs.

Achieving this goal is crucial for:

- Compliance with environmental regulations (e.g., Euro 6, EPA Tier 3 standards).

- Reduces the vehicle's impact on air pollution and climate change.

- Enhances sustainability, making HEVs a viable alternative in the transition to greener mobility.

However, the following challenges must be considered:

- ICE emissions vary with driving cycles, making real-time optimization complex.

- Trade-offs between emissions and fuel economy—minimizing one may increase the other.

- Battery sourcing and recycling must also be considered in lifecycle emissions.

#### 2.1.3. Maintaining battery health

The battery is a critical component of HEVs, and its long-term health must be preserved through effective SOC management, avoiding deep discharges and overcharging, and regulating battery temperature to prevent degradation.

The following advantages highlight the relevance of considering battery health in the EMS:

- Ensures consistent performance and longer battery lifespan.

- Reduces the need for costly battery replacements.

- Enhances the efficiency and reliability of HEV powertrains.

Nevertheless, Key difficulties arise from the fact that:

- Battery degradation is influenced by multiple factors, including temperature, charge-discharge cycles, and SOC variations.

- Regulating SOC to balance longevity and energy availability is difficult under real-world conditions.

- Heat management is crucial, especially in high-power demand situations (e.g., fast acceleration, regenerative braking).

#### 2.1.4. Ensuring vehicle performance

The EMS must ensure that the vehicle meets performance expectations in terms of:

- Acceleration and torque delivery for responsiveness.

- Top speed capabilities aligned with vehicle design.

- Smooth transitions between power sources for a seamless driving experience.

The impact of this objective is evident in:

- Enhancing driver satisfaction and safety.

- Maintaining competitive performance compared to conventional vehicles.

- Reducing torque interruptions, improving ride comfort.

On the other hand, several challenges arise due to:

- Performance demands may conflict with fuel economy and emissions reduction goals.

- High power requests may lead to increased battery discharge rates, reducing longevity.

- Balancing power delivery from ICE and EM to avoid lag in response time is crucial.

#### 2.1.5. Optimizing energy efficiency

Energy efficiency in HEVs refers to the optimal use of all available energy sources (fuel, electricity, and regenerative braking) to maximize overall powertrain efficiency. This includes:

- Optimizing the energy flow between ICE, EM, and battery.

- Using predictive control methods to anticipate energy demands based on road conditions and driver behavior.
- Implementing real-time adaptation strategies for dynamic conditions (e.g., urban vs. highway driving).

This objective is important because it provides following advantages:

- Enhance vehicle fuel economy and driving range.

- Reduce emissions and operational costs.

- Improve HEV longevity by efficiently managing powertrain components.

At the same time, some complexities are associated with this objective, including

- Energy efficiency depends on real-time power distribution, which requires advanced control algorithms.

- Computational complexity increases with more variables.

- Predicting driving conditions accurately for proactive energy management is challenging.

## 2.2. Key parameters in EMS optimization

An effective EMS in HEVs must optimize key parameters such as:

- Power split ratio: Determines the distribution of power between the ICE and the EM to ensure efficient energy utilization based on driving conditions.

- Battery state of charge: Maintains the battery within optimal operating limits to prevent degradation, maximize energy recovery through regenerative braking, and support electric propulsion.

- Engine operating points: Ensures the ICE operates in high-efficiency regions by optimizing torque and speed while implementing start-stop strategies to minimize fuel consumption.

- EMusage: Manages EM operation for propulsion, regenerative braking, and standby modes to optimize energy efficiency and battery longevity.

- Waste heat recovery: Utilizes technologies like ORC to convert ICE waste heat into usable power, improving overall energy efficiency.

## 2.3. Constraints in energy management model

The optimization of HEVs EMS is subject to several constraints to ensure safe, efficient, and regulatory-compliant operation. These constraints help balance performance, energy efficiency, and component longevity while maintaining drivability across various driving conditions. The commonly considered constraints are cited as follows:

- Battery SOC limits: The battery SOC must remain within a predefined range (e.g., 20% to 80%) to prevent overcharging or deep discharging, which can accelerate degradation and reduce battery lifespan.

- Power demand fulfillment: The total power supplied by the ICE and EM must always meet the vehicle's instantaneous power demand to meet the vehicle's required power at any given moment.

- Component operating limits: The ICE and EM must function within their maximum power, torque, and speed constraints to prevent excessive wear, overheating, or potential failure.

- Thermal management: The temperature of the battery, ICE, and power electronics must remain within safe operating limits to avoid thermal runaway, efficiency losses, or system shutdowns.

- Driving cycle adaptability: The EMS must dynamically adjust power distribution strategies to optimize efficiency across various driving conditions, including urban, highway, and mixed driving cycles.

- Emissions compliance: The ICE operation must align with emission regulations (e.g., Euro 6, EPA standards) to limit the release of greenhouse gas emissions and particulate matter, requiring optimized engine control and hybrid power balancing.

By enforcing these constraints, the EMS ensures optimal energy distribution, system longevity, and environmental compliance while maintaining the vehicle's performance and drivability.

# 3. Reinforcement learning

Reinforcement learning is a machine learning research area involving autonomous agent training to make decisions in an unknown environment to maximize a desired cumulative reward. The agent performs a large number of trial and error experiences by being either recompensed or punished to learn the best behavior. The most popular approach for RL problem modeling is the Markov decision process (MDP) [17] (see Fig. 2).

## 3.1. State space

The state variables indicate the actual system status, and they are the basis for choosing actions. They can be represented by a vector with a finite dimension containing system dynamic parameters such as power/torque demand, battery state of charge, remaining fuel, bus voltage, vehicle acceleration, or vehicle velocity. Variables are sampled into discrete values to build a state grid, considering all feasible combinations. Increasing variables number in the state space $S$ will improve the environment description accuracy and reliability [26]. Increasing sampling frequency to get higher discretization may give better results in terms of control effectiveness [27]. However, with a large state dimension, the computational burden will be higher and the convergence will be slowed down. This is because the number of states often increases exponentially as the number of state variables increases. This is known as the “curse of dimensionality” issue. Thus, a trade-off should be made in choosing the appropriate state parameters with a reasonable discretization degree. For example, in [28], authors settled for load power $P_{load}$ and battery state of charge SOC for a hydrogen electric train power system. Table 2 summarizes common state variables used in HEVs energy management strategies.

## 3.2. Action space

The agent takes actions to control the system while maximizing the rewards. Actions encompass any control we want to train the agent to perform in the best way. It can be motor torque, motor speed, fuel cell power reference, or change of fuel cell delivered power. Similar to state space, action space $A$ is discrete. For example, if we consider the EMS controlling the fuel cell power change, the action space is given as an $n$-dimensional tuple of potential variations of the fuel cell power output [43]. Fig. 3 illustrates an example of the interaction between the agent and state space for a hybrid electric bus application [44]. Actions are represented by the bus control signals (engine torque, rotational speed). The environment includes vehicle state variables (speed, acceleration) and also traffic information such as link speed, which is used in traffic flow modeling and refers to other vehicles speed on a given road section. Table 3 presents an overview of the commonly used state variables in HEV energy management strategies.

## 3.3. Rewards and return

The agent objective is to maximize not only the instantaneous received reward, but also the long-term cumulative expected rewards and known as return ($G_t$). Indeed, the return is represented by a discounting sum of expected future rewards. An adaptive discount factor can be
<figure><img src="image_5.png" /><figcaption>Fig. 2. Agent-environment interaction illustration.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 3. Example of an agent-environment interaction [44].</figcaption></figure>

adopted, beginning with a lower $\delta$ and increasing it along the learning stage speeds up the learning process [52].

$$G_t = R_{t+1} + \delta R_{t+2} + \delta^2 R_{t+3} + \dots = \sum_{0}^{T} \delta^k R_{t+1+k} \quad (1)$$

where $\delta$ refers to the discount factor and $T$ is the termination time.

The choice of the reward function is critical because it affects convergence speed and control quality of the reinforcement learning EMS [21]. It is generally expressed using the objective function $F$ of the targeted optimization problem:

$$R_t = -F = -[\beta_1 \cdot f_1 + \beta_2 \cdot f_2 + \dots + \beta_n \cdot f_n] \quad (2)$$

where $f_1 \sim f_n$ represent optimization goals like fuel and electricity consumption [53], battery SOC deviation [54], and energy sources degradation cost [43]. $\beta_1$ - $\beta_n$ represent goal weights that can be set by tuning. Indeed, the reward function is usually normalized to [0, 1] using tanh() function as demonstrated in (3). This normalization improves the convergence speed [54].

$$R_t = -\tanh[F] \quad (3)$$

Besides objective functions, optimization constraints must also be considered in the reward function. Constraints are usually used to keep power sources operating within their limits, such as minimum/ maximum output power, SOC range, and power change rate. If any constraint is violated during training or any generated state/action is infeasible, the agent will be penalized by introducing a negative reward as a punishment ($R_t = -1$). On the contrary, if the agent performs very well achieving perfect results (zero SOC deviation [28]), an additional

recompense could be given ($R_t = R_t + 1$) as presented in (4).

$$R_t = \begin{cases} -\tanh[F] + 1 & \text{when recompensing} \\ -1 & \text{when punishing} \\ -\tanh[F] & \text{else} \end{cases} \quad (4)$$

Many studies focus on minimizing fuel or hydrogen consumption by penalizing their usage in their reward function [30,32]. Maintaining the battery SOC within a desired range is essential for ensuring battery health and optimal vehicle performance. To achieve this, reward functions typically incorporate penalty terms for deviations from the target SOC [55], effectively balancing energy usage between the engine and the battery. To improve the durability and operational lifespan of vehicle components, some reward functions incorporate penalty terms for actions that induce accelerated degradation [56], such as excessive engine start-stop cycling, sustained high-power demands on the battery, and rapid fluctuations in fuel cell power output [57].

Developing an effective reward function is essential for creating RL-based EMS in for HEVs. The reward function plays a critical role in guiding the learning process, directly impacting the agent's ability to make optimal decisions about energy distribution. Various studies have investigated methodologies for designing reward functions to improve EMS performance. For instance, Hu et al. [58] conducted a thorough recent review of RL applications in EMS for electric vehicles with multiple power sources. They examined various reward function configurations and their influence on training efficiency, highlighting that a well-crafted reward function is crucial for meeting control objectives and ensuring optimal energy utilization. This study emphasizes the importance of carefully designing reward functions that are specifically tailored to the objectives of the EMS. A well-structured reward function not only speeds up the learning process but also ensures that the actions taken by the RL agent are in line with broader goals of energy efficiency
<figure><img src="image_7.png" /><figcaption>Fig. 4. Common rewards, states, and action variables used in RL-based EMSs for HEVs.</figcaption></figure>

and system sustainability [59]. Table 4 provides a comprehensive summary of the rewards frequently employed in energy management strategies for HEVs, and Fig. 4 illustrates the most commonly used reward, state, and action variables in HEV RL-based energy management studies.

## 3.4. Value functions

Reinforcement learning algorithms generally require value function estimations. The state-value function, denoted as $v_\pi(s)$, is the expected return when being in state $s$ and following the policy $\pi$. Similarly, the action-value $q_\pi(s, a)$ is the expected return from choosing a particular action $a$ in a given state $s$ while following the policy $\pi$:

$$ egin{cases} v_{\pi}(s) = \mathbb{E}_{\pi} \left( \sum_{0}^{T} \delta^{k} R_{t+1+k} | S_{t} = s \right) = \mathbb{E}_{\pi}(G_{t} | S_{t} = s) \\ q_{\pi}(s, a) = \mathbb{E}_{\pi} \left( \sum_{0}^{T} \delta^{k} R_{t+1+k} | S_{t} = s, A_{t} = a \right) \\ = \mathbb{E}_{\pi}(G_{t} | S_{t} = s, A_{t} = a) \end{cases} \quad (5) $$

During learning, the agent calculates the average of all returns that have followed a given state $s$. This average will converge to $v_\pi(s)$ (Fig. 5(a)). In the same way, returns average for every action performed in every state will converge to the action-values $q_\pi(s, a)$ (Fig. 5(b)).

Using learning experiences, reinforcement learning algorithms always seek to match policies that provide the highest returns over the long term. These optimal policies $\pi^*$ have expected returns that are greater than or equal to all other policies for all states [17]. They have common best state-value and action-value functions ($v^*(s)$ and $q^*(s, a)$):

$$ egin{cases} v^*(s) = v_{\pi^*}(s) = \underset{\pi}{\operatorname{argmax}} v_\pi(s) \\ q^*(s, a) = q_{\pi^*}(s, a) = \underset{\pi}{\operatorname{argmax}} q_\pi(s, a) \end{cases} \quad (6) $$

# 4. RL-based energy management

RL has been successfully applied to HEV energy management in the literature. Fig. 6 shows the implementation process, starting from real driving cycles gathering to online implementation. Hereafter, we will explain the concept of RL algorithms that have been widely adopted in HEV EMS applications.

## 4.1. Q-learning (QL)

Q-learning algorithm is a common algorithm proposed by Watkins [66]. It is a model-free RL strategy, which seeks to estimate the optimal policy based on maximizing the expected state-action value function $Q$, also denoted $q(s, a)$, for all possible state-action pairs. The approximation of $Q$ is done regardless of the followed policy, which makes Q-learning an off-policy-based algorithm. For each time step, the interaction between the agent and the environment yields an update in the $Q$ table, as indicated in the following expression:

$$ Q(S_t, A_t) \leftarrow (1 - \alpha)Q(S_t, A_t) + \alpha \left[ R_{t+1} + \delta \max_a Q(S_{t+1}, a) \right] \quad (7) $$

where $\alpha \in (0, 1]$ is the learning rate. It established how much is the agent learning from new experiences rather than using directly old acquired estimates. If $\alpha = 0$, the agent will just exploit prior knowledge without learning anything. If $\alpha = 1$, the agent will ignore all its prior learning and consider only the current estimation. A fixed learning rate is commonly used in practice, for instance, $\alpha = 0.1$.

The term $(1 - \alpha)Q(S_t, A_t)$ is the current $Q$ value weighted by the learning rate. The quantity $\max_a Q(S_{t+1}, a)$ refers to the maximum estimate that can be obtained in state $S_{t+1}$, discounted by $\delta$. Indeed, the learning rate serves as a weighting factor between the observed $Q$ value and the new estimated one.

Eq. (8) shows that action selection is usually performed using an $\epsilon$-greedy method, which has proven better performance than a
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 5. Illustration of (a) the state-value function $v^*(s)$ back-up and (b) the action-value function $q^*(s, a)$ back-up.</figcaption></figure>

<table><caption>**Table 2**

Common state variables in HEV RL-based EMSs.</caption><thead><tr><th>State variable</th><th>Importance for EMS</th><th>Ref.</th></tr></thead><tbody><tr><td>SOC/SOH</td><td>Critical for preserving battery health and longevity</td><td>[29]</td></tr><tr><td>Power demand</td><td>Guides allocation between power sources, ensuring smooth power transitions.</td><td>[30]</td></tr><tr><td>Vehicle speed</td><td>Helps adapt energy management to driving conditions (e.g., urban vs. highway driving).</td><td>[31]</td></tr><tr><td>Acceleration</td><td>Assesses vehicle dynamics for energy distribution.</td><td>[32]</td></tr><tr><td>Engine speed</td><td>Monitors engine performance for optimization.</td><td>[19]</td></tr><tr><td>Engine coolant temperature</td><td>reflects engine efficiency, aids in reducing cold start emissions, and plays a crucial role in thermal management</td><td>[33]</td></tr><tr><td>Torque demand</td><td>Reduce mechanical stress on drivetrain components by adjusting motor torque based on demand.</td><td>[34]</td></tr><tr><td>Gear position</td><td>Used to optimize gear shifting strategies.</td><td>[35]</td></tr><tr><td>EMspeed</td><td>Manages interaction between motor and engine.</td><td>[36]</td></tr><tr><td>Fuel consumption</td><td>Minimizes fuel usage and emissions by optimizing ICE operation.</td><td>[37]</td></tr><tr><td>Traffic/terrain information</td><td>Allows to anticipate changes in driving conditions and adjusts power allocation based on traffic congestion and surface conditions.</td><td>[38]</td></tr><tr><td>Road grade</td><td>Adjusts strategy based on uphill or downhill driving and helps in energy recovery.</td><td>[39]</td></tr><tr><td>Yaw rate</td><td>Enhances stability control and optimizes energy distribution during turns</td><td>[40]</td></tr><tr><td>Battery temperature</td><td>Prevents overheating and ensures safe and efficient battery operation.</td><td>[33]</td></tr><tr><td>Travel distance</td><td>Aids in improving vehicle range and optimizing energy usage for long trips.</td><td>[36]</td></tr><tr><td>Environmental temperature</td><td>Regulates passenger comfort with energy-efficient climate control.</td><td>[41]</td></tr><tr><td>Energy price</td><td>minimizes operating costs and adapts to dynamic energy prices, ensuring cost-effective energy management.</td><td>[42]</td></tr></tbody></table>

<table><caption>**Table 3**

Common action variables in HEV RL-based EMSs.</caption><thead><tr><th>Action variable</th><th>Importance for EMS</th><th>Ref.</th></tr></thead><tbody><tr><td>Change in engine power</td><td>Adjust the engine's power output to balance fuel efficiency and battery SOC.</td><td>[45]</td></tr><tr><td>Battery power</td><td>determine energy allocation between the battery and other power sources to maintain SOC and minimize degradation.</td><td>[46]</td></tr><tr><td>Supercapacitor power</td><td>Enhances power management by stabilizing load fluctuations.</td><td>[47]</td></tr><tr><td>FC power</td><td>Controls hydrogen consumption and battery charging, impacting overall energy efficiency.</td><td>[48]</td></tr><tr><td>Engine torque</td><td>Controls power delivery from the internal combustion engine to optimize efficiency and performance.</td><td>[49]</td></tr><tr><td>ICE control signal</td><td>Regulates engine operation, optimizing fuel consumption and energy distribution.</td><td>[37]</td></tr><tr><td>Engine speed and torque</td><td>Crucial for optimizing engine operation and fuel efficiency.</td><td>[18]</td></tr><tr><td>Rotating speed</td><td>Regulates energy transfer between components to optimize performance.</td><td>[19]</td></tr><tr><td>Acceleration</td><td>Influences energy demand and fuel efficiency by modifying driving dynamics.</td><td>[35]</td></tr><tr><td>Gear position</td><td>Ensures smooth gear transitions for better performance and energy savings.</td><td>[38]</td></tr><tr><td>Change in gear ratio</td><td>Helps optimize the powertrain efficiency by selecting the appropriate gear.</td><td>[45]</td></tr><tr><td>Drive mode</td><td>Switches between different energy modes (e.g., electric, hybrid, fuel) to optimize efficiency.</td><td>[50]</td></tr><tr><td>Change in steering Angle</td><td>Enhances vehicle control and energy consumption optimization.</td><td>[35]</td></tr><tr><td>Throttle control</td><td>Regulates fuel injection and air intake to optimize fuel consumption and power output.</td><td>[51]</td></tr><tr><td>Heat exchanger output power</td><td>Regulates thermal management to maintain optimal operating temperatures.</td><td>[33]</td></tr><tr><td>Cooling capacity</td><td>Control the air conditioning system seeking passenger comfort</td><td>[41]</td></tr></tbody></table>

greedy strategy [17]. The block diagram of the Q-learning algorithm is depicted in Fig. 7, and its pseudo-code is summarized in Algorithm 1.

$$A_t = \begin{cases} \max_a Q(S_t, a) & \text{with a probability of } 1 - \epsilon \\ a & \text{a random action with a probability of } \epsilon \end{cases} \quad (8)$$

It is well known that Q-learning achieves the optimal value function with certainty [67]. In addition, when choosing an appropriate learning rate $\alpha$, the algorithm shows an exponential convergence with trials number [68]. However, it suffers from a poor convergence rate,

especially if the discount factor $\delta$ approaches 1. Speedy Q-learning (SQL) algorithm is then proposed to deal with this limitation [69]. A decreasing learning rate is adopted instead of a fixed one when updating $Q$ function ($\alpha_t = 1/(t+1)$) while involving two estimates $Q(t-1)$ and $Q(t)$ related to previous and current time steps. Indeed, it was proven that SQL leads to a fast learning process [69], however, it increases the algorithm complexity.
<table><caption>**Table 4**
Different rewards used in RL-based EMS for HEVs.</caption><thead><tr><td>Reward</td><td>Significance for EMS</td><td>Ref.</td></tr></thead><tbody><tr><td>Fuel consumption</td><td>Minimizes fuel usage, improving overall energy efficiency and reducing operating costs and emissions.</td><td>[60]</td></tr><tr><td>SOC target error</td><td>Ensures the battery operates within an optimal SOC range, extending its lifespan.</td><td>[59]</td></tr><tr><td>Battery SOC change rate</td><td>Prevents abrupt SOC fluctuations, protecting battery health and improving longevity.</td><td>[61]</td></tr><tr><td>Energy loss</td><td>Reduces energy dissipation improving overall system efficiency.</td><td>[62]</td></tr><tr><td>Equivalent fuel consumption</td><td>Provides a unified cost metric to balance fuel and electricity consumption.</td><td>[53]</td></tr><tr><td>Fuel cost</td><td>Optimizes economic performance by minimizing operational costs related to fuel usage.</td><td>[39]</td></tr><tr><td>FC power change rate</td><td>Avoids excessive power variations, enhancing the durability and efficiency of the fuel cell.</td><td>[57]</td></tr><tr><td>Power loss</td><td>Minimizes resistive and conversion losses, improving energy utilization efficiency.</td><td>[63]</td></tr><tr><td>Energy saving</td><td>Enhances overall energy efficiency by reducing unnecessary consumption.</td><td>[31]</td></tr><tr><td>FC and battery degradation</td><td>Enhances system reliability and efficiency by mitigating long-term performance degradation of energy sources.</td><td>[64]</td></tr><tr><td>Battery aging</td><td>Reduces battery degradation by controlling charge/discharge cycles, extending battery lifespan.</td><td>[65]</td></tr><tr><td>Engine efficiency</td><td>Ensures the powertrain operates at its optimal efficiency, minimizing energy losses and fuel consumption.</td><td>[35]</td></tr><tr><td>Gear ratio deviation</td><td>Ensures optimal gear shifting for efficient energy use and smooth operation.</td><td>[45]</td></tr><tr><td>Distance error</td><td>Maintains a safe following distance to improve driving safety and efficiency in car-following scenarios.</td><td>[45]</td></tr></tbody></table>

**Algorithm 1: Q-learning algorithm pseudo-code**

<figure><img src="image_10.png" /></figure>

## 4.2. Double Q-learning

Any practical estimation method can lead to cumulative errors due to the fact that real values are unknown. Those errors can be

<figure><img src="image_11.png" /><figcaption>**Fig. 6.** RL-based EMS implementation process.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 7.** Block diagram of the Q-learning algorithm.</figcaption></figure>

caused by non-stationary behaviors, environmental noises, or approximations [70]. Indeed, by involving the max operator to identify the next state value in Q-learning, the action value may be strongly overestimated. That induces a propagating positive bias when selecting the highest value as an estimate, consequently, algorithm performance will be penalized.

Double Q-learning was proposed in [71] to overcome the overestimation issue using two estimators ($Q_A$ and $Q_B$ functions) instead of
<figure><img src="image_13.png" /><figcaption>Fig. 8. Block diagram of the DQL algorithm.</figcaption></figure>

a biased one, and by separating the selection of the estimator from the calculation of the estimate. As shown in (8), every Q-function is updated by involving the other Q-function with equal probabilities. When updating $Q_A$ in a state $s$, we observe the action $a^*$ that maximizes $Q_A(a^* = \max_a Q_A(s, a))$, then we use the $Q_B$ estimate for that selected action $Q_B(s, a^*)$, instead of adopting $Q_A(s, a^*)$ value, and vice versa. The block diagram of the DQL algorithm is shown in Fig. 8. Regarding Q-functions updating (Algorithm 1, line 14), it will be done according to (9) as depicted in Algorithm 2. This is regarded as an unbiased estimation since $Q_B$ is actualized in an identical environment, with different experience samples than $Q_A$ ones. It is worth noting that the double Q-Learning approach requires twice the memory space without increasing the computational burden.

$$ \left\{ \begin{array}{l} Q_A(S_t, A_t) \leftarrow (1-\alpha)Q_A(S_t, A_t) + \\ \qquad \alpha \left[ R_{t+1} + \delta \max_a Q_B(S_{t+1}, \max_a Q_A(s, a)) \right] \\ \\ Q_B(S_t, A_t) \leftarrow (1-\alpha)Q_B(S_t, A_t) + \\ \qquad \alpha \left[ R_{t+1} + \delta \max_a Q_A(S_{t+1}, \max_a Q_B(s, a)) \right] \end{array} \right. \quad (9) $$

### 4.3. Sarsa

The name Sarsa is derived from the transitions happening in Markov sequences from one state-action pair to another giving rise to the quintuple $(S_t, A_t \to R_{t+1}, S_{t+1}, A_{t+1})$ [72]. Starting from $(S_t, A_t)$ pair, and after performing again $A_{t+1}$ from $S_{t+1}$ (following an $\epsilon$-greedy method), then $Q(S_t, A_t)$ will be updated by $Q(S_{t+1}, A_{t+1})$ instead of using $\max_a Q_A(S_{t+1}, a)$ as an off-policy would do:

$$ Q(S_t, A_t) \leftarrow (1 - \alpha)Q(S_t, A_t) + \alpha [R_{t+1} + \delta Q(S_{t+1}, A_{t+1})] \quad (10) $$

Unlike Q-learning, Sarsa is an on-policy RL algorithm. $q_\pi$ is estimated using the actually followed policy $\pi$, not the supposedly optimal policy as Q-learning does. When Q-learning approach uses max operator in updating, it is supposed that the completely greedy policy is followed ($\epsilon = 0$). However, this is not the case since the agent does not actually perform only greedy actions. Thus, Q-learning will converge to an optimal solution assuming that a greedy policy will be followed after training. In other words, the target policy is not the same as the

<figure><img src="image_14.png" /><figcaption>**Algorithm 2: Double Q-learning algorithm pseudo-code**</figcaption></figure>

behavior policy. On the other hand, Sarsa will converge to an optimal solution assuming that we continue to follow the same policy used in the learning stage. On-policy algorithms are generally simpler, and they have lower variance and faster convergence compared to off-policy methods. It is worth noting that if action selection is greedy ($\epsilon = 0$), Sarsa and Q-learning would be the same algorithms. Table 5 presents a comparative analysis of Sarsa, Q-learning, and Double Q-learning algorithms in the context of energy management for HEVs, highlighting their key characteristics, advantages, and limitations.

### 4.4. Energy management with RL methods

In a study dealing with a hybrid electric tracked vehicle EMS [73], it has been reported that Q-learning led to less fuel consumption reaching 2.4% economy compared to the stochastic dynamic programming (SDP) approach, and up to 22.7% when compared to a rule-based strategy. In a similar study [51], speedy Q-learning algorithm has shown a fuel consumption gain of 2.9% and 5.5% compared to SDP and Q-learning, respectively. Bai et al. [74] applied Q-learning algorithm for a hybrid fixed-wing UAV powered by a battery and internal combustion engine to reduce emissions and conserve battery SOC. The reported results regarding fuel consumption showed a gain of 2.18% relative to the power tracking method in a cruise flight mission, and about 4.4% in a terrain tracking flight mission. Q-learning also showed better response to the required power high fluctuations improving UAV maneuverability.

In [30], authors implemented an improved Q-learning-based EMS intended for HEV powered by fuel cell, battery, and ultracapacitor. Equivalent fuel consumption minimization was adopted as an objective inspired by the ECMS approach. In addition, a transition probability matrix (TPM) obtained based on more than 40 thousand real driving cycles was introduced in the algorithm. The results outline an outstanding fuel economy compared to ECMS ranging from 7.1% to 26% depending on the driving cycle. A TPM is a stationary Markov chain, which is used to model the required power using real power profiles. It provides the probabilities $p_{i,j}$ of moving from a state $S_i$ to another one $S_j$ in one time step:

$$ \left\{ \begin{array}{l} p_{i,j} = \frac{M_{i-j}}{M_i} \\ M_i = \sum_j M_{i-j} \end{array} \right. \quad (11) $$
<table><caption>**Table 5**
Comparison of Sarsa, QL, and DQL for HEV energy management.</caption><thead><tr><th>Aspect</th><th>Sarsa</th><th>Q-learning</th><th>Double Q-learning</th></tr></thead><tbody><tr><td>Type</td><td>On-policy</td><td>Off-policy</td><td>Off-policy</td></tr><tr><td>Exploration/Exploitation</td><td>Balances exploration and exploitation based on the current policy.</td><td>Focuses on exploitation by always choosing the best action.</td><td>Reduces overestimation of Q-values, improving exploitation.</td></tr><tr><td>Convergence</td><td>Converges to the optimal policy while considering exploration.</td><td>Converges to the optimal policy but may overestimate Q-values.</td><td>Converges more reliably by mitigating Q-value overestimation.</td></tr><tr><td>Computational Complexity</td><td>Moderate</td><td>Moderate</td><td>Slightly higher due to maintaining two Q-tables.</td></tr><tr><td>Robustness to Noise</td><td>More robust to noise due to on-policy learning.</td><td>Less robust to noise due to greedy action selection.</td><td>More robust than Q-learning due to reduced overestimation.</td></tr><tr><td>Advantages</td><td>Better than QL in online learning. Conservative: Avoid dangerous optimal paths. Adapts to costly real systems training. Safer in real-time applications due to on-policy learning.</td><td>Simpler to implement. Faster convergence in stable environments. Temporal difference updating. Flexible and adapts to multi-decisions tasks. Preferred in offline learning than Sarsa.</td><td>Reduces overestimation bias. More reliable in noisy environments.</td></tr><tr><td>Limitations</td><td>Learns a near optimal policy: Decaying ε in ε-greedy strategy may be a solution. Instability due to correlation between consecutive experiences. Slower convergence due to on-policy updates.</td><td>Slow convergence. Overestimation. Curse of dimensionality. Not adapted to continuous tasks. Deterministic: Cannot learn stochastic policies.</td><td>Higher computational cost. Slightly slower convergence than Q-learning. Increased memory requirement</td></tr><tr><td>Adaptability to HEV EMSs</td><td>Adapts well to dynamic driving conditions by considering current policy.</td><td>Suitable for static or predictable driving conditions.</td><td>Better suited for dynamic and uncertain driving conditions.</td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>**Fig. 9.** Power transition probability matrix illustration [62].</figcaption></figure>

where $M_{i-j}$ is the number of transitions from the load power $P_{load}^i$ to $P_{load}^j$, when $M_i$ represents the total number of transitions performed from $P_{load}^i$. Fig. 9 illustrates an example of TPM for an HEV application [62]. The TPM establishment process is demonstrated in [21].

Online RL-based strategy updating is studied in [62]. The proposed algorithm performs online updates to the Q-learning policy according to current driving conditions. Kullback-Leibler divergence rate threshold is adopted to decide whether the new TPM is sufficiently different from the learning one. When this is the case, the strategy is updated involving the new TPM. The results showed a 16.8% decrease in energy loss relative to the rule-based EMS.

Xu et al. [53] proposed a Q-learning parametric analysis considering HEV fuel economy. It was found that training experiences choice has an impact on learning performance. Selecting the best experiences from available data is more beneficial than learning from every experience. This result was also confirmed in another study [39], where near-optimal driving cycles control decisions are used for initialization accelerating learning convergence. Actions and states sensitivity analysis indicates that less fuel is consumed when action resolution

is improved or state number is increased while increasing state resolution is detrimental to fuel economy. The authors also reported that increasing the exploration rate $\epsilon$ up to 10% reduces fuel economy. However, beyond 10%, the opposite effect appears. In another sensitivity study [75], results outline the influence of learning rate $\alpha$ and discount factor $\delta$. Faster policy convergence is obtained when increasing $\alpha$ from 0.01 to 0.1 and $\delta$ from 0.1 to 0.9.

A comprehensive performance analysis of reinforcement learning (RL)-based energy management strategies PHEVs and fuel cell electric vehicles (FCEVs) was conducted in [76], focusing on algorithm efficiency, perception, and decision granularity, hyperparameter optimization, and reward function design. It was found that off-policy algorithms such as Q-learning outperform on-policy algorithms such as SARSA in achieving fuel efficiency and balanced battery consumption. The importance of fine granularity in state and action spaces was highlighted for precision in energy management decisions despite the increased computational complexity. The proper tuning of hyperparameters and designing reward functions based on the overall SOC variation have been shown to enhance the stability and safety of the EMS. A multi-objective intelligent energy management system was presented in [29] for HEVs utilizing multiagent reinforcement learning. The system aims to optimize multiple conflicting objectives such as fuel efficiency, emission reduction, and battery longevity. By employing a multi-agent framework, each agent focuses on a specific objective, learns, and adapts its strategies to a cooperative environment. The results showed a good improvement in the overall energy management of HEVs, demonstrating the effectiveness of multiagent reinforcement learning in balancing diverse performance metrics [64].

A real-time energy management strategy for plug-in hybrid electric vehicles (PHEVs) was proposed in [61], which utilized double-delay Q-learning and model prediction control. This hybrid method integrates the advantages of reinforcement learning and model-based predictive control to achieve an optimal energy distribution between the engine and electric motor. The proposed system adapts to real-time changes in driving conditions, enhances fuel efficiency, and minimizes emissions. The simulation results confirm the ability of the proposed strategy to maintain a balance between the performance and computational efficiency. An optimal energy management strategy for PHEVs was developed in [63] using ensemble reinforcement learning with exploration-to-exploitation ratio control. The ensemble learning approach aggregates multiple reinforcement learning models, each with
varying degrees of exploration-exploitation trade-offs, to achieve a more robust and effective energy management system. The results showed significant improvements in fuel efficiency and emissions reduction, emphasizing the potential of ensemble methods to improve the performance of PHEVs under various driving conditions.

A model-based offline RL framework was developed in [55] for energy management in HEVs. This solution addresses the inadequacies and safety concerns of conventional RL methods by integrating uncertainty-aware models. The offline training process enables the system to learn from extensive historical data, whereas the uncertainty-aware approach guarantees reliable performance in real-time applications, leading to improved fuel efficiency and reduced emissions.

Ensemble supervisory control approach was tested for the first time in [77] by using multi-agent RL. Authors combined multiple strategies namely: 2/4 states-based Q-learning, ECMS, and thermostatic strategy. The results showed that the four agents combination performs better than the best single agent (4 states Q-learning) with 3.2% less fuel consumption. However, not all multi-agent combinations showed better results than the best single agent. Agents selection is thus fundamental.

Shen et al. [26] proposed an improved double Q-learning approach targeting hybrid UAV flight time extension. Their idea consists on emphasizing the most valuable state-action pairs in Q tables. These particular areas are refined during the learning process to lock the optimal policy earlier accelerating the convergence. The obtained results show that the strategy can achieve more than 90% of fuel economy compared to the optimal solution given by dynamic programming. A hybrid energy management strategy was introduced in [32] for HEVs, which combines offline initialization with online fine-tuning using RL. This approach uses offline data to initialize the energy management model, which is then fine-tuned in real time to adapt to dynamic driving conditions. The proposed method enhances learning efficiency and system performance, leading to improved fuel efficiency and reduced emissions through effective energy management. An online energy management strategy for fuel cell vehicles was developed in [57], using fuzzy Q-learning and road condition recognition. The proposed method dynamically adjusts the energy distribution based on real-time road conditions, improves fuel consumption, and reduces system degradation. The integration of fuzzy logic enhances the adaptability and robustness of the algorithm, making it suitable for diverse driving environments and conditions.

Imitation learning (IL) is an emerging field for energy management. It can be used with and without RL methods for EV EMS. In IL algorithms, generative adversarial imitation learning (GAIL), behavior cloning (BC), and inverse RL (IRL) are mostly used. However, BC is considered more effective due to its simplicity, fast implementation, and low computational requirements for GAIL and IRL [78,79]. Since this paper is focused on reviewing RL and DRL methods, papers involving IL with RL and DRL methods for EV EMS are only considered. A new concept of imitation learning was introduced in [56] to address concerns related to RL, such as computational complexity, large data training, the rigorous need to anticipate future operating conditions, and irregular control actions in the preliminary stages of exploration. Firstly, offline global optimization is conducted to explore power allocation paths under various driving conditions. Subsequently, battery depletion boundaries corresponding to driving distances are incorporated to reduce the state space, within which the optimal trajectory informs the RL training. In contrast to RL, imitation learning bypasses the use of a reward function for agent updates through environmental interactions. Instead, it involves direct demonstrations from an expert (global optimization) to the agent, allowing the agent to learn the optimal policy by emulating the decisions made by the expert [31]. Finally, the trained agent with an optimal policy is deployed in the online controller and verified within the testing environment. This strategy demonstrates better optimality, adaptability, and robustness performance than a standard Q-learning agent that has undergone typical training.

The comparative analysis of RL-based HEV EMSs is presented in Table 6. It provides the analysis of states, actions, and reward objectives of RL-based HEV EMSs, which helps the readers in better understanding the role of different states, actions, and rewards in terms of effective implementation of RL-based HEV EMSs. The main contributions of these listed papers are provided in this subsection. However, further improvements are possible by considering the effects of depth of discharge on battery life, emissions reduction with ICE-based HEVs, learning rate, hyperparameters tuning, range factor, real-time testing, and customer privacy issues with the use of charging infrastructures.

## 5. Deep reinforcement learning algorithms

RL-based strategies can perform an efficient energy management task in general. However, when dealing with continuous state/action variables or a high-dimensional state space, the discretization results in an exponential increase of the Q function matrix size. The Q table will be ridiculously large and impractical. Exploring all possible actions with an $\epsilon$-greedy strategy would take a very long time leading to a high computational burden and low convergence performance [18]. This refers to the aforementioned “curse of dimensionality” issue. For example, if we consider only 4 variables with only 100 grids each, the resulting state number is 100 million ($100^4$).

In addition, $\epsilon$-greedy strategy could lead to a poor exploration task. For example, given an environment with 10 possible actions $a_1 - a_{10}$, and a Q-learning agent following an $\epsilon$-greedy technique with $\epsilon = 0.1$. Supposing that at a given state $s_t$, the highest Q value corresponds to $a_1$ where $Q(s_t, a_1) = 1.01$, $Q(s_t, a_2) = 1.00$, and $Q(s_t, a_3) = 0.01$. The agent will select $a_1$ with a probability of 0.91 (90% of greedy selection and 1% of random selection), when the other actions have only a chance of 1% to be chosen, even though Q values for $a_1$ and $a_2$ are very close. The situation is even worse if we consider that $a_3$ and $a_2$ are equiprobable even though $Q(s_t, a_2)$ is a hundred times better.

To overcome classic RL algorithms limitations, a deep Q network (DQN) based on a deep neural network (DNN) is introduced to replace the Q table for Q values approximation (Fig. 10). In contrast to the Q matrix, any continuous variation in the state variables leads to a change in the DNN output decision. Thus, state discretization is avoided allowing larger state space. That also allows considering more environment variables improving accuracy and control effectiveness without increasing the computational burden. In addition, DNN performs well as an approximation function, which enables policy faster convergence. Indeed, the larger the state space is, the more superior the DNN will be to standard Q-Learning in terms of convergence. When the state is overly complex (millions of possible states such as in Atari games), Q-learning gets almost impossible to converge due to time and hardware limitations. Because it will require a very large memory and a very long training time to be able to enumerate and explore all possible states. However, DNN can generalize states and elaborate a function between state and action. It does not require enumeration, it learns information implied in the states. Even if some states were not explored during training, as long as DNN has been trained to learn the relationship over other similar states, it can still generalize and produce the appropriate Q-value. Convergence is therefore significantly faster.

### 5.1. Deep Q-learning (DQL)

In DQL algorithm, a DNN is implemented to replace the Q matrix for value function Q estimation following policy $\pi$ as follows:

$$Q(S_t, A_t, \theta) \approx Q_{\pi}(S_t, A_t) \qquad (12)$$

where $\theta$ refers to the DNN parameters (weights and biases), and $Q(S_t, A_t, \theta)$ represents the output corresponding to state-action pair $(S_t, A_t)$. Using DQN, a vector of all possible action values $Q(S_t, A_t, \theta)$ can be produced for each given state $S_t$ under policy $\pi$. DQN is trained with a Q-learning algorithm using stochastic gradient descent to update
<table><caption>**Table 6**
Comprehensive analysis of RL-based EMS for HEVs.</caption><thead><tr><th>Ref.</th><th>Method</th><th>States</th><th>Actions</th><th>Rewards</th><th>EV type</th></tr></thead><tbody><tr><td>[51]</td><td>Q-learning</td><td>Battery SOC, generator speed</td><td>Throttle control</td><td>Fuel consumption, SOC target error</td><td>HEV</td></tr><tr><td>[30]</td><td>Q-learning</td><td>Demand power, battery and ultracapacitors SOCs, vehicle speed</td><td>Battery power, fuel cell power</td><td>Hydrogen consumption, SOC target error</td><td>FCHEV</td></tr><tr><td>[62]</td><td>Q-learning</td><td>Demand power, battery SOC, capacitor voltage</td><td>Battery current</td><td>Energy loss of hybrid energy storage</td><td>HEV</td></tr><tr><td>[53]</td><td>Q-learning</td><td>Vehicle speed, demand torque, gear number</td><td>Generator torque</td><td>Equivalent fuel consumption</td><td>HEV</td></tr><tr><td>[39]</td><td>Q-learning</td><td>Vehicle speed, road grade, destination time, battery SOC, available charging gain</td><td>ICE power supply</td><td>Fuel cost</td><td>HEV</td></tr><tr><td>[32]</td><td>Online RL</td><td>Speed, acceleration, battery SOC</td><td>Engine demand torque</td><td>Fuel consumption, SOC target error</td><td>HEV</td></tr><tr><td>[55]</td><td>Uncertainty-based Q-learning</td><td>Battery SOC</td><td>Equivalent fuel consumption</td><td>SOC target error</td><td>HEV</td></tr><tr><td>[57]</td><td>Fuzzy Q-learning</td><td>Demand power, battery SOC</td><td>Fuel cell power</td><td>Hydrogen consumption, SOC target error, rate of power change of fuel cell</td><td>FCHEV</td></tr><tr><td>[29]</td><td>Nash Q-learning</td><td>Velocity, battery and ultracapacitor SOCs</td><td>Engine rotational speed, torque</td><td>Fuel consumption, SOC constraint violation</td><td>HEV</td></tr><tr><td>[63]</td><td>Ensemble Q-learning</td><td>Battery SOC, link power</td><td>Engine generator control signal</td><td>Power loss, battery SOC reference error</td><td>HEV</td></tr><tr><td>[26]</td><td>Double Q-learning</td><td>Demand power, speed, battery SOC</td><td>Control signal</td><td>Fuel consumption</td><td>HEV</td></tr><tr><td>[61]</td><td>Double-Delay Q-Learning</td><td>Battery SOC, required power</td><td>Battery power</td><td>Fuel consumption, battery SOC change rate</td><td>HEV</td></tr><tr><td>[56]</td><td>Imitation Q-Learning</td><td>Demand power, battery SOC</td><td>Engine power</td><td>Equivalent fuel consumption</td><td>PHEV</td></tr><tr><td>[31]</td><td>Imitation Q-Learning</td><td>Required torque, vehicle speed</td><td>Supercapacitors power</td><td>Energy saving and battery aging</td><td>BEV</td></tr></tbody></table>

<figure><img src="image_16.png" /><figcaption>**Fig. 10.** DQN illustration.</figcaption></figure>

its parameters $\theta$. The objective is to minimize the loss function $L$ which is represented by the mean-squared temporal-difference (TD) error between the target $Q$ value $y_t(\theta)$ and the predicted one $Q(S_t, A_t, \theta)$, like in a typical regression problem.

$$ \left\{ \begin{aligned} y_t(\theta) &= R_{t+1} + \delta \max_a Q(S_{t+1}, a, \theta) \\ L(\theta) &= \mathbb{E} \left[ \left( y_t(\theta) - Q(S_t, A_t, \theta) \right)^2 \right] \end{aligned} \right. \quad (13) $$

However, the issue is that the same parameters $\theta$ are used to estimate both target and $Q$-value. Consequently, we will have a high correlation between the changing parameters $\theta$ of the DQN noted $Q_\theta$ and the target $Q$ value. This implies that at every training time step, $Q$ values change but also target values do. We are approaching the target, but the target is also shifting. This results in high oscillations in training. Thus, a

separate fixed $Q$-target network is introduced using fixed parameters $\theta^-$ which are obtained from some previous iterations [80]. This fixed DQN noted $Q_\theta^-$ is adopted to avoid dependency between target $Q$ values and $\theta$ improving the algorithm stability. $Q_\theta^-$ is periodically updated by transferring $\theta$ from $Q_\theta$ for each $C$ steps.

$$ \left\{ \begin{aligned} y_t(\theta^-) &= R_{t+1} + \delta \max_a Q(S_{t+1}, a, \theta^-) \\ L(\theta^-) &= \mathbb{E} \left[ \left( y_t(\theta^-) - Q(S_t, A_t, \theta^-) \right)^2 \right] \end{aligned} \right. \quad (14) $$

An important element of the DQL algorithm training is the experience replay. Agent's experience represented by a tuple $(S_t, A_t, R_{t+1}, S_{t+1})$ is saved in a data set named replay memory at every time step. In practice, the replay memory has usually a limited length, only $N$ last experiences are stored. Instead of replacing $S_t$ with the new state $S_{t+1}$ for the next update as we do in standard Q-learning, a new
<figure><img src="image_17.png" /><figcaption>Fig. 11. Deep Q-learning EMS training illustration [81].</figcaption></figure>

**Algorithm 3: DQL algorithm pseudo-code**

Inputs: $\delta \in [0, 1], e \in [0, 1], rand_e \in [0, 1], Q_\theta$ action value DQN, $\theta, \theta^{-}(Q_\theta$ and $Q_{\theta^{-}}$ parameters), D (recall memory), N (batch size).

1 Initialize $Q_\theta$ and $Q_{\theta^{-}}$ networks
2 for every learning episode: do
3 while state $\neq$ final state do
4 for each time step: do
5 observe the current state $S_t$
6 if rand_e < e then
7 $A_t = \max_a Q(S_t, a, \theta)$ /* exploitation */
8 else
9 $A_t$ = a random action /* exploration */
10 end
11 Perform action $A_t$, observe $S_{t+1}$ and $R_{t+1}$
12 Store experience in D
13 Sample a random batch from D:
14 $\sum_{i=1}^{N}(S_i, A_i, R_{i+1}, S_{i+1})$
15 for t = 1 : N do
16 $y_t = R_{t+1} + \delta \max_a Q(S_{t+1}, a, \theta^{-})$
17 $L(\theta) = E\left[\left(y_t - Q(S_t, A_t, \theta)\right)^2\right]$
18 end
19 Perform gradient descent to minimize $L(\theta)$
20 Update $\theta$
21 end
22 every C steps: reset $Q_{\theta^{-}} = Q_\theta$
23 end
24 end

disconnected transitions batch sample is randomly selected from the replay memory to train the DQN by applying gradient descent on the selected batch. This task is called experience replay, it is repeated every time step. Since Q-learning is an output policy-based approach, it does not necessarily have to be performed along sequential paths [17]. DQL algorithm pseudo-code is provided in Algorithm 3.

Experience replay technique has several advantages. The main one is obtained by breaking the correlation between successive tuples and then reducing the variance. In addition, by eliminating dependencies between consecutive samples $\theta$, a source of instability will be removed. Fig. 11 shows an illustration of a deep Q-learning EMS diagram [18].

## 5.2. Policy gradient algorithms

The Q-learning process seeks to identify the Q-value function for each state-action pair. This learning scheme is built on the concept

of determining the quality of all feasible actions, and then deciding based on this knowledge. Q-learning, therefore, aims to get unbiased and full knowledge of all possible decisions. But That is also its major disadvantage since it needs a large number of trials for each possible state. The policy gradient algorithm performs more robustly. Instead of attempting to estimate the value of all actions, it simply predicts the best action to take in each state. This can be very effective in continuous or large state spaces where getting value's argmax on all possible actions would be prohibitively expensive. The policy makes actions with high rewards more likely or the opposite. It keeps what works well and discards what does not. The model (neural network) will directly generate the selection probability for each action $p(a, S_t, \theta)$, avoiding additional calculations related to the Q function. Gradient policy algorithm learns a parameterized policy that can choose actions without checking the value function. The training objective is to increase policy performance, thus, a scalar performance measure $J(\theta)$ is defined. To update $\theta$, the algorithm seeks then to maximize the derivative of $J(\theta)$ by applying gradient ascent, the opposite of gradient descent:

$$ \theta_{t+1} = \theta_t + \alpha \nabla \widehat{J}(\theta) \quad (15) $$

where $\nabla \widehat{J}(\theta)$ is a stochastic estimate, which approximates $J(\theta)$ gradient [17]. It is worth noting that policy gradient algorithms perform all the updates retrospectively after the end of each episode, like Monte Carlo algorithms, and differently than the temporal-difference approach. The performance measure can be defined as the return $G_t$, which represents the total accumulated rewards from time t including all future rewards until the end of the episode. That can be defined by the state value function $v_{\pi_\theta}$ for the policy $\pi_\theta$, and evaluated in the initial state of the episode $S_0$ [17]:

$$ J(\theta) = v_{\pi_{\theta}}(S_0) \quad (16) $$

The gradient calculation is detailed in [17] and is given by:

$$ \nabla J(\theta) = G_t \nabla_{\theta} \ln[\pi(A_t|S_t, \theta_t)] \quad (17) $$

where $\pi(A_t|S_t, \theta_t)$ is the probability of selecting action $A_t$ in state $S_t$ given parameter vector $\theta_t$.

The policy gradient algorithm is an on-policy approach. Its learning process is based only on the state-action transitions performed by the actual followed policy. This means that all data sets are used only once. Thus, experience replay and replay memory are not required as in DQN. Once the model is trained, policy parameters $\theta$ are updated at the end of each episode (not after every time step) using gradient ascent, as shown in Algorithm 4.
<table><caption>**Algorithm 4: Policy Gradient algorithm pseudo-code**</caption><thead><tr><th>Inputs:</th><th></th></tr></thead><tbody><tr><td>α, δ, θ<sub>t</sub>, π(a|s, θ).</td><td></td></tr><tr><td>1</td><td>Initialize policy parameters θ</td></tr><tr><td>2</td><td>for every learning episode e<sub>k</sub>: do</td></tr><tr><td>3</td><td>&nbsp;&nbsp;&nbsp;&nbsp;Run the episode and generate consecutive experiences respecting π(a|s, θ):</td></tr><tr><td>4</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∑<sup>T−1</sup><sub>i=0</sub> (S<sub>i</sub>, A<sub>i</sub>, R<sub>i+1</sub>, S<sub>i+1</sub>)</td></tr><tr><td>5</td><td>&nbsp;&nbsp;&nbsp;&nbsp;while state ≠ final state of e<sub>k</sub> do</td></tr><tr><td>6</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for each time step t of e<sub>k</sub>: do</td></tr><tr><td>7</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;G<sub>t</sub> ← ∑<sup>T−1</sup><sub>i=0</sub> δ<sup>i</sup> R<sub>i+1</sub></td></tr><tr><td>8</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;θ<sub>t</sub> ← θ<sub>t</sub> + αδ<sup>i</sup> G<sub>t</sub> ∨ ln[π(A<sub>t</sub>|S<sub>t</sub>, θ<sub>t</sub>)]</td></tr><tr><td>9</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end</td></tr><tr><td>10</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end</td></tr><tr><td>11</td><td>end</td></tr></tbody></table>

### 5.3. Actor-critic methods

In policy gradient algorithms, the agent learns directly a policy using the performance metric which is maximized in the course of learning. There is no necessity to know the actual value of each action. However, that leads to a certain policy dependency on the performance measure. We take an example to explain this issue. Assuming that the performance measure is set as the overall reward across a 100-steps episode. We suppose that all steps yield a reward of +10 except step 83, which gave -100. The accumulated return for the full trajectory will be high and the policy will average all actions taken in this episode as good. Consequently, the agent will keep selecting the same bad action in step 83 since it does not have knowledge about its actual value. Thus, our performance metric cannot detect that pitfall. To overcome this drawback, the agent can also learn state and action values in parallel with learning the policy. This combination is called actor-critic method. Two sub-agents are involved to learn in parallel. The Actor learns directly the policy as in the policy gradient method while critic learns the state-action value as in Q-learning (Fig. 12). The update rule is then changed substituting the return $G_t$ by $Q(S_t, A_t)$ leading to a more complex architecture:

$$ \theta_{t+1} = \theta_t + \alpha \left( Q(S_t, A_t) \nabla_{\theta} \ln[\pi(A_t|S_t, \theta_t)] \right) \quad (18) $$

The update process is performed every time step as in TD methods. The agent does not need to wait till the end of the episode to do it as in the policy gradient approach.

Hybrid RL methods have been developed to enhance HEV energy management by combining policy gradient and Actor-Critic approaches [82]. Policy gradient methods optimize stochastic policies in high-dimensional action spaces, while Actor-Critic structures improve learning stability by evaluating actions. DDPG algorithm integrates these strengths, using a deterministic policy with target networks for stability. This allows efficient handling of continuous control tasks, making DDPG a robust solution for optimizing energy distribution in HEVs. Fig. 13 illustrates the concept of the Deep Deterministic Policy Gradient (DDPG) algorithm applied to HEV energy management, highlighting its actor-critic framework and deterministic policy for optimizing energy distribution [35]. In this context, another Hybrid RL approach that benefits from both policy-based and value-based reinforcement learning has been employed in the literature developing twin delayed deep deterministic policy gradient (TD3) algorithm, which is an improvement over the DDPG algorithm [83]. Fig. 14 depicts the framework of the proposed strategy. Twin Critic Networks consist of two separate critic networks (Critic 1 and Critic 2) that estimate the Q-values ($q_1$ and $q_2$) for the selected action a. The presence of twin critics helps mitigate overestimation bias in value estimation. The Target Networks include a target actor and target critics (Target Critic 1 and Target Critic 2) to compute stable target Q-values. The action $a'$ is

perturbed by adding noise $\epsilon$ to encourage exploration. The minimum of the two Q-values, min($q'_1, q'_2$), is selected to compute the target, reducing overestimation errors.

Table 7 presents a comparative analysis of Deep Q-Learning, Policy Gradient, and Actor-Critic algorithms, outlining their respective advantages and limitations in the context of energy management strategies for hybrid electric vehicles (HEVs).

### 5.4. Energy management with DRL methods

Deep Q-learning algorithms have been widely adopted for HEVs EMS. In a study dedicated to a hybrid electric bus [18], the DQL was compared to Q-learning regarding training, SOC deviation, and state variables. It has been reported that the DQL training process was 3.6 times faster than Q-learning (2.3 h against 8.3 h). Q-learning algorithm could not learn quickly due to the lack of information about bus acceleration and speed. At the same time, it was unable to effectively manage the connection between keeping battery SOC and enhancing fuel consumption. Both algorithms were able to maintain initial SOC. However, DQL leads to a 5.6% better fuel economy achieving 89% of dynamic programming optimality. In similar studies, DQL was also implemented in [81] for an HEV decreasing equivalent fuel consumption by 2.5% relative to the rule-based strategy. In addition, DRL-based online learning architecture was proposed to allow the EMS to learn and adapt in real-time to varying driving conditions, enhancing its applicability in HEVs [84]. In [85], DQL-based EMS is developed for a battery electric vehicle (BEV) powered by high-power (HP) and high-energy (HE) batteries to minimize energy loss and enhance safety. Results show a 96.5% decrease in training time and 54.5% in computation time, achieving 6 kJ less energy loss relative to the QL-based EMS.

DNN hyperparameters have to be appropriately adjusted since they directly impact DQL performance. Their tuning can be done manually. However, it will be time-consuming. Some strategies are proposed in the literature to do this task such as random search [86]. However, such a method is computationally expensive. In this context, Kong et al. [87] proposed Bayesian optimization (BO) as an efficient strategy for hyperparameters tuning. Simulation results have shown good DQL learning performance compared to the random search method. In addition, the implemented DQL for a supercapacitor/battery-powered vehicle performed slightly better than a near-optimal rule-based strategy in terms of energy consumption and battery life preservation. In [88], hyperparameter optimization has been carried out using a new greedy method to accelerate on-the-fly learning. It was proved that the proposed approach is more than 5 times faster than BO leading to about 80% less energy consumption during training. However, BO was slightly more accurate.

In a recent study [89], an accelerated DQL algorithm with prioritized replay was implemented for HEV EMS. In standard DQL, experience replay allows the agent to recall and reuse previous experiences uniformly sampled from a replay memory, neglecting their importance. The prioritized experience replay approach emphasizes significant transitions involving them with higher frequency and therefore learns more effectively. The EMS has included a specific function to measure the priority, set transitions, and update the significance weights. The reported hardware-in-the-loop simulation results have shown up to 10.6% in fuel economy and a gain of up to 86.7% in computational cost relative to a standard DQL.

A double deep Q-learning (DDQL) algorithm was introduced in [19] for a hybrid electric tracked-vehicle EMS. Like double Q-learning, DDQL has prevented the EMS from falling into overestimation. The reported results demonstrated that DDQL outperformed the standard DQL by 7.1% in fuel economy, reaching 93.2% of Dynamic programming optimality. In addition, the DDQL maintained battery SOC considering different initial SOC.
<figure><img src="image_18.png" /><figcaption>Fig. 12. Actor-Critic training diagram.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Fig. 13. DDPG illustration for HEV energy management [35].</figcaption></figure>

In [36], authors explored a deep deterministic policy gradient (DDPG) strategy, which is an actor-critic off-policy approach that does not need any discretization for state/action spaces. It uses experience replay memory and involves two copies of both actor and critic networks; one copy is for training, and the other one is fixed and updated periodically. The algorithm optimality in terms of fuel economy was validated, achieving an average deviation of only 3.5% from deterministic dynamic programming. DDPG approach was also investigated in [44] for a series-parallel plug-in hybrid electric bus. Real traffic information gathered by using sensors is employed to enhance

learning performance. The proposed DDPG outperformed the standard Q-learning algorithm in terms of fuel consumption and showed better generality to new driving cycles, which were very different from training ones. An improved DDPG algorithm is proposed in [82]. The conjunction of two techniques was proven to achieve effective learning: a critic network with a dual configuration is adopted, and pre-training is performed before the learning process. A dual architecture has two streams for separately estimating the state value and advantage of every action related to that state. The two values are then aggregated to provide a good estimate of the state-action value. The pre-training is
<figure><img src="image_20.png" /><figcaption>Fig. 14. TD3 algorithm illustration [83].</figcaption></figure>

<table><caption>**Table 7**
Comparison of DRL algorithms for HEV Energy Management.</caption><thead><tr><th>Aspect</th><th>Deep Q-Learning (DQL)</th><th>Policy Gradient (PG)</th><th>Actor-Critic (AC)</th></tr></thead><tbody><tr><td>Type</td><td>Off-policy</td><td>On-policy</td><td>On-policy</td></tr><tr><td>Exploration/ Exploitation</td><td>Balances exploration and exploitation using experience replay and ε-greedy methods.</td><td>Relies on stochastic policies for exploration, which can be less efficient.</td><td>Combines value-based and policy-based strategies for better exploration.</td></tr><tr><td>Convergence</td><td>Converges to optimal Q-values but may suffer from overestimation.</td><td>Converges to a local optimum, often with high variance.</td><td>More reliable than PG but can still be unstable (step sizes have to be carefully tuned).</td></tr><tr><td>Computational Complexity</td><td>High due to deep neural networks and experience replay.</td><td>Moderate, but high variance increases computational cost.</td><td>High due to dual networks (actor and critic).</td></tr><tr><td>Robustness to Noise</td><td>Robust due to experience replay, but sensitive to hyperparameters.</td><td>Less robust due to high variance in policy updates.</td><td>More robust than PG but still sensitive to hyperparameters.</td></tr><tr><td>Advantages</td><td>Good stability thanks to experience replay. Powerful computing performance. Handles high-dimensional state spaces. No curse of dimensionality.</td><td>Adapts to stochastic environments and continuous control. Effective in high-dimensional or continuous action spaces. Robust with good convergence.</td><td>Combines strengths of value- and policy-based methods. Adapts to stochastic policies. Independent of performance metrics.</td></tr><tr><td>Limitations</td><td>Higher per-sample variance than Sarsa. Complicated computations. Hyperparameter tuning issues. Less adapted to high-dimensional action spaces.</td><td>High variance and slow convergence. Unrealistic rewards due to global averaging. Typically converges to a local optimum. Difficult to stabilize model parameters.</td><td>Requires extensive hyperparameter search. Low stability and generalization. Time-consuming learning.</td></tr><tr><td>Adaptability and use case in HEVs</td><td>Suitable for complex energy management tasks with high-dimensional state spaces, such as optimizing power split and battery SOC deviation.</td><td>Effective for continuous control tasks, such as torque distribution and engine power.</td><td>Ideal for tasks requiring a balance between exploration and exploitation, such as adaptive energy management in dynamic driving conditions.</td></tr></tbody></table>

carried out using optimal experience samples obtained by DP. These ideal samples are produced and copied to the replay memory at the initial stage.

In a recent study [90], rule-based methods were integrated with the DDPG algorithm to improve energy management in HEVs equipped with an ORC waste heat recovery system. The authors introduced a DRL-embedded rule-based (DRLERB) EMS, which combines the reliability of rule-based control with the optimization potential of deep reinforcement learning. The proposed EMS significantly enhances fuel economy and overall system efficiency by utilizing waste heat recovery via ORC. The key findings reveal that the DRLERB-EMS reduces energy consumption by 4.31% compared to traditional rule-based strategies when evaluated on the C-WTVC driving cycle. Furthermore, the real-time feasibility of the approach was confirmed through a rapid control prototype (RCP) implementation, showcasing its practical applicability in real-world HEV energy management. These outcomes underscore the effectiveness of DRL-based EMS solutions in managing multi-source energy distribution and optimizing hybrid powertrains with integrated waste heat recovery systems. In this context, Wang et al. [91] proposed a similar study for an HEV-ORC system. Simulation results show that

the DRL-based EMS reduces fuel consumption by 2% compared to rule-based methods while ensuring higher engine and motor efficiency, stable ORC power output, and balanced battery management.

Distributed DRL architecture was investigated in [92] by proposing an asynchronous advantage actor-critic (A3C) algorithm. Indeed, the Q value when considered solely does not always reflect the action importance, and that may be confusing. Thus, the agent in A3C learns the advantage $A(S_t, A_t)$ of choosing a specific action compared to others instead of only learning its Q value.

$$A(S_t, A_t) = Q(S_t, A_t) - V(S_t) \quad (19)$$

where $V(S_t)$ represents the state value function. In addition, multiple agents are used to learn the same policy but updated asynchronously, which increases the learning process speed and robustness. Simulation results showed a 4 times superior convergence rate and better fuel economy achieved by A3C compared to the standard DQL. In another recent study [50], an integrated thermal and energy management system for a multimode-linked PHEV was developed using an actor-critic method. The suggested agent uses sophisticated methods like delayed policy updates, bounded double Q-values, and a spatiotemporal difference (STDP) framework. In comparison to the standard DRL-based EMSs, the HIL outcomes show that the proposed method achieves a
<figure><img src="image_21.png" /><figcaption>Fig. 15. Multi-agent DDPG algorithm illustration [49].</figcaption></figure>

close to DP performance achieving 93.7% of its efficacy, while yielding significant reductions in fuel consumption from 2.2% to 9.6% across a temperature range from 15 °C to -15 °C.

A multi-agent deep reinforcement learning-based cooperative control strategy was introduced in [49] to address the complex multi-objective optimization challenges inherent in hybrid electric vehicles (HEVs). By utilizing numerous agents, each of which is responsible for a specific control aspect, the strategy optimizes both fuel economy and emission reduction. The proposed framework coordinates the actions of these agents and ensures that the overall system performance improves. The fundamental framework of the proposed approach is shown in Fig. 15 [49].

An optimized cluster-based energy management system was introduced in [42] using a double DQN method, with a focus on vehicle-to-everything (V2X) technologies and peer-to-peer energy trading. The objective aims to optimize energy efficiency and prosumer comfort without compromising performance. The integration of V2X communication enables the system to make data-driven decisions based on real-time information from other vehicles and the infrastructure. The double DQN algorithm was designed to adapt to changing energy demands, ensuring efficient energy distribution and enhanced prosumer satisfaction.

In [93], a novel physics-informed exploration method is combined with a safe reinforcement-learning strategy for HEV EMS. The proposed TD3 method ensures the safety and reliability of the EMS by integrating physical constraints and domain knowledge into the exploration process. This approach enhances the efficiency of fuel usage, decreases emissions, and ensures secure operation of the system across diverse driving scenarios, effectively tackling real-world issues encountered in vehicle energy management. In a similar study [94], an advanced EMS for FCHEVs was developed using TD3 algorithm improving training efficiency, while enhancing SOC stability, and minimizing usage costs by integrating fuel cell and battery degradation models.

An improved EMS was introduced in [38] for heavy-duty HEV using a soft actor-critic method with a discrete-continuous hybrid action space. This approach enables a more adaptable and detailed regulation of the energy distribution between the engine and electric motor, resulting in a substantial enhancement of the fuel efficiency and a reduction

in emissions by optimizing the performance of the powertrain in response to varying driving conditions. A comprehensive control strategy was introduced in [35], which utilized a combination of DDPG and DQN methods for HEVs EMS. High-definition maps were used to provide precise information on the vehicle surroundings to achieve more accurate and efficient energy management. Optimization of power allocation between the engine and electric motor was achieved, resulting in enhanced fuel efficiency and decreased emissions. In [95], a degradation-adaptive EMS was developed for FCHEVs in which the rule-based DDPG method was used to particularly accommodate the current state of health (SOH) of the powertrain. TEMS strategy was adjusted according to the degradation level of the fuel cell and battery, thereby ensuring optimal performance and longevity. The integration of rule-based control and DDPG showed significant enhancements in fuel efficiency and the reliability of vehicle components.

In [96], an adaptive soft deep Q-learning algorithm was used to propose an optimal energy management strategy for multisource FCHEVs based on driving behavior. The proposed method considers the driver's behavior and preferences and adapts an energy management strategy to optimize fuel efficiency, emission reduction, and driving comfort. This approach improves vehicle performance under diverse driving conditions and individual driving styles. An energy management strategy was proposed in [37] for PHEVs using Bayesian optimization and a soft actor-critic algorithm. This hybrid approach utilizes the strengths of Bayesian optimization for parameter tuning and the soft actor-critic algorithm for continuous control. The proposed strategy optimizes the power distribution between the engine and electric motor, enhancing fuel efficiency and reducing emissions.

An energy management strategy was developed in [48] for optimal charge depletion in plug-in FCHEV using multi-constraint TD3 with fuzzy logic. The proposed approach considers several constraints, including fuel efficiency, emission reduction, and component longevity, to achieve optimal energy distribution. The multi-constraint TD3 ensures that the energy management strategy adapts to varying driving conditions and energy demands, resulting in substantial enhancements in the overall vehicle performance. An uncertainty-based DQN method was presented in [34] to optimize energy management strategies in hybrid electric vehicles (HEVs). The proposed approach addresses the
uncertainties in the driving conditions and energy demands, thereby improving the robustness and efficiency of the EMS. It incorporates action advising, which facilitates the learning process and achieves superior fuel economy and emission reduction compared with conventional reinforcement learning algorithms.

A hybrid energy management strategy was introduced in [47] for battery/supercapacitor HEVs using the DDPG method. It incorporates locality-sensitive hashing (LSH) and prioritized experience replay (PER) to improve exploration and enhance learning efficiency, resulting in better system performance. This method has been successful in reducing battery power consumption and current fluctuations in vehicle energy management. A continuous RL-based energy management strategy was introduced in [40] for hybrid electric-tracked vehicles, considering steering resistance. DDPG algorithm optimizes the power allocation between the engine and electric motor, resulting in improved fuel economy and robust performance. This study highlights the ability of the algorithm to adapt to the unique operational conditions of tracked vehicles, with improved efficiency and performance. A supplementary learning control strategy was introduced in [97] for energy management of HEVs at scale using the TD3 method. This approach enhances traditional EMSs by incorporating additional learning layers that can adapt to large-scale operational data. It improves fuel efficiency, reduces emissions, and enhances the scalability of energy management systems for extensive vehicle fleets, thereby demonstrating its potential for widespread applications.

In [46], an imitation learning environment integrated with a DRL EMS is developed for a FCHEV application. For the purpose of overall operating cost minimization, the EMS considers fuel usage, FC deterioration, and battery aging. Proximal Policy Optimization (PPO) is adopted as a DRL strategy, while Behavioral Cloning is the offline global optimization method (expert policy). The proposed approach reaches 95.59% of the optimal operational cost while reducing training time by 51.69% compared to standalone PPO and increasing total reward by 4.57%. A dual actor-imitator DRL framework is presented in [98] that integrates imitative expert experience to enhance interpretability and training efficiency in smart charging for grid-connected EVs. The aim is to derive optimal charging strategies using a V2G cost-benefit analysis and develop an experienced pool to guide the learning process for parameterized DQN. The results showed the effectiveness of the proposed dual actor-imitator DRL framework in improving V2G benefits while mitigating battery aging.

An incentive learning-based energy management strategy was developed in [99] for hybrid energy storage systems in electric vehicles, comprising batteries and supercapacitors. It integrates an incentive reward based on the supercapacitor SOC and vehicle acceleration to encourage optimal supercapacitor usage under high load conditions in order to reduce battery degradation and power losses. PPO is combined with generalized advantage estimation (GAE) and layer normalization (LN) to further improve learning stability and convergence. Additionally, a velocity transfer probability surface is used to pre-train the RL agent on diverse driving cycles, ensuring adaptability to unfamiliar conditions. The results showed that the method significantly outperformed existing RL methods, reducing battery and power loss costs by 5.8%–13.8% and 11.7%–38.8%, respectively, as compared to the offline dynamic programming method. In [100], a combined GAIL and DDPG-based adversarial imitation RL strategy was proposed for hybrid energy storage management in EVs. Dynamic programming is used to generate optimal policy for the RL agent as initial guidance during training, which ensures fast convergence and learning performance of the RL agent. This proposed method provides dynamic imitation weights to smoothly transit from imitation based on expert guidance to RL self-exploration and increases adaptability to different driving scenarios. The results showed that the training speed using the proposed method was 42.60% higher compared with classical RL, the reward was improved by about 15.79%, and the costs of battery capacity loss were lower by 5.1%–12.4% across various typical driving cycles.

Tang et al. [45] have suggested a vision & DRL-based hierarchical control technique that satisfies vision-based recognition, distance evaluation, and energy management in a car-following scenario. Real-time distance between two cars is measured once the leading car is identified in the driving image. The hybrid powertrain is therefore allowed to perform multiobjective control by using a deep Q-network to learn both the energy management and car-following control policies. The results showed an acceptable fuel economy and a significant level of computational efficiency on embedded systems and hold hope for realistic vehicle control.

Most EMSs are based on an offline learning process, which can only be performed using existing driving data under a simulation framework. However, in real-time implementation, the environment will not necessarily be similar to the training one. HEV will deal with new driving conditions that may be very different from the training ones. Consequently, EMS performance will be affected. Thus, online learning is proposed in the literature to overcome this limitation [81]. Online learning disconnects the training process from action execution. The DQN controller chooses an appropriate action for the HEV using an *ε*-greedy strategy and stores the state-action transitions. These stored data are used by another remote or onboard computer for real-time training. The computer updates the DQN parameters periodically. Indeed, three key elements make online learning different from offline learning: (i) adaptation to changing circumstances by adjusting EMS parameters, (ii) real-time training separation from control actions, and (iii) higher efficiency due to fast optimal control learning considering fresh experiences. It is worth mentioning that online learning can be preceded by an offline one using driving patterns extracted from past experiences [62]. Therefore, learning efficiency will be improved.

Transition probabilities between vehicle states are recursively updated to adapt to dynamic driving conditions, unlike static models with fixed probabilities. To detect significant environmental changes, Kullback-Leibler (KL) divergence measures shifts in transition probability matrices. If it exceeds a threshold, the policy update process is initiated. First, the power transition matrix is updated to reflect the latest observed dynamics. Subsequently, the policy is retrained using these revised transition probabilities, ensuring that decision-making remains aligned with the current operational environment. Finally, the new policy is seamlessly integrated into the system without restarting the training process, allowing continuous learning without disrupting vehicle performance. This real-time adaptability is essential for maintaining optimal energy efficiency and power allocation across varying driving cycles.

An innovative online-learning adaptive energy management strategy was proposed in [101] for HEVs using the Dyna framework, which integrates real-time learning and planning, allowing the energy management system to dynamically adapt to varying driving conditions. The Dyna framework combines model-based and model-free reinforcement-learning techniques, providing a robust solution for enhancing fuel efficiency and reducing emissions in various driving scenarios. Hu et al. [102] experimentally demonstrated the effectiveness of online DRL-based EMS regarding fuel economy. In addition, the learning approach has proven to be effective in achieving optimality and real-time implementability. Results also show that the pre-training phase contributes to reducing learning time because the EMS learns from some similar characteristics in different driving cycles. In another study [28], a DQL algorithm is proposed to control the fuel cell/battery power system for a train application. Total Cost of Ownership (TCO) is considered besides fuel economy and SOC conservation. The approach changes the action space dimension in the course of learning depending on the SOC deviation. HIL experiments showed that the proposed strategy reached 96% of optimality compared with the dynamic programming method while keeping battery SOC. Fig. 16 illustrates an online learning process for four deployed HEVs linked to a remote center via cloud computing.
<figure><img src="image_22.png" /><figcaption>Fig. 16. Online learning concept illustration.</figcaption></figure>

The comparative analysis of DRL-based HEV EMSs is presented in Table 8. It provides the analysis for states, actions, and reward objectives of DRL-based HEV EMSs. However, further improvements are possible by considering the effects of depth of discharge on battery life, emissions reduction with ICE-based HEVs, learning rate, hyperparameters tuning, range factor, real-time testing, and customer privacy issues with the use of charging infrastructures.

DRL algorithms have shown good generality to new driving conditions that differ from training ones. When encountering new similar control tasks or using a new HEV type, the agent has to be retrained from zero to learn the optimal policy for adapting to the new environment, leading to a tedious training time. In addition, it is difficult to generate a good and extensive dataset due to the cost of data acquisition [103]. In this context, transfer learning (TL) may be combined with DRL to solve this issue, decreasing energy consumption and overcoming the problem of small data availability. TL is a machine learning approach that exploits shared knowledge to perform different tasks in related fields [104]. The previous knowledge will be exploited to accelerate learning, and the ancient policy will be adapted instead of learning from zero. Most of the parameters can be stored in the new neural network, and the model will be recycled to build the new problem solution (Fig. 17). The other parameters will be randomly initialized and updated with RL. Then, fine-tuning is applied to adjust all parameters using a relatively small number of driving cycles related to the new domain. Therefore, the training time will be significantly reduced.

In a recent study [105], authors could transfer energy management knowledge between two HEVs with significantly different structures. When applied to DDPG, the performance of TL was evaluated considering the five metrics defined in [106] to assess the potential of the transferring task. The metrics include fuel consumption, convergence efficiency, and generalization ability. The obtained results have demonstrated that the proposed method has achieved an average gap of 70% from the baseline in terms of convergence efficiency. In [107], DDPG and TL were connected in an adaptive EMS for an HEV. A two-level control scheme is implemented to generate the EMS. The upper level applies the DDPG strategy for training the EMS at various velocity intervals, while the lower level uses the TL method to transform the pre-trained neural networks for a new driving cycle. Authors in [108]

investigated exploration strategies for a DDPG-based transfer learning method. The reported experimental results have shown that parameter space noise achieved good performance in terms of adaptation and stability.

DRL methods have shown great potential for optimizing energy management in HEVs. However, these methods typically require large amounts of data and extensive training periods, which can be challenging in real-world HEV applications. Researchers have explored several strategies to adapt DRL methods to the HEV context to overcome these challenges. Data augmentation has emerged as a critical solution to compensate for the lack of real-world driving data and improve the robustness of RL-based EMS. By artificially expanding the training dataset, data augmentation techniques allow RL agents to experience a broader range of driving conditions, thereby enhancing their generalization ability [109]. Common approaches include domain randomization [110], which introduces stochastic variations in simulated environments to improve adaptability, and trajectory augmentation, which generates synthetic driving trajectories based on probabilistic models [111]. Authors in [112] used DP to extend their dataset for training their EMS dedicated to PHEVs. Another approach is meta-learning, which enables RL agents to quickly adapt to new driving conditions with limited data by leveraging prior experience [113]. A recent study proposed a meta-energy management strategy (meta-EMS), enabling rapid adaptation to new HEV scenarios with minimal training [114].

### 5.5. Convergence and adaptability of DRL algorithms for HEV energy management

DRL algorithms have demonstrated distinct levels of convergence speed, energy optimization effectiveness, and adaptability to real-world scenarios in energy management tasks. Therefore, selecting the most appropriate DRL algorithm remains an open question, as each method offers different advantages and limitations. In a recent study [58], a comparison of several DRL algorithms including Q-learning, DQN, Double DQN, TD3, DDPG, PPO, and SAC (Soft Actor-Critic) was conducted, focusing on their learning stability, energy optimization performance, and generalization capabilities. The analysis reveals that TD3 and PPO stand out as top performers due to their balanced convergence speed
<table><caption>**Table 8**
Comprehensive analysis of DRL-based EMS for electric vehicles.</caption><thead><tr><th>Ref.</th><th>Method</th><th>States</th><th>Actions</th><th>Rewards</th><th>EV type</th></tr></thead><tbody><tr><td>[18]</td><td>Deep Q-learning</td><td>Battery SOC</td><td>Engine speed and torque</td><td>Fuel consumption</td><td>HEV</td></tr><tr><td>[19]</td><td>Double deep Q-learning</td><td>Battery SOC, demand power, speed, acceleration, angular speed</td><td>Rotating speed</td><td>Fuel consumption rate, SOC target error</td><td>HEV</td></tr><tr><td>[85]</td><td>Deep Q-learning</td><td>Battery power, battery SOC, battery temperature, vehicle speed</td><td>Control signal</td><td>Energy loss</td><td>BEV</td></tr><tr><td>[36]</td><td>DDPG</td><td>Travel distance, SOC, SOC error, speed, acceleration, engine power</td><td>Change in engine power</td><td>Fuel consumption rate, SOC target error</td><td>HEV</td></tr><tr><td>[34]</td><td>Uncertainty-based DQN</td><td>Engine speed, torque, battery SOC</td><td>Engine power</td><td>Fuel consumption, SOC target error</td><td>HEV</td></tr><tr><td>[47]</td><td>LSH-PER-DDPG</td><td>Demand power, battery SOC, super capacitor SOC</td><td>Battery power, supercapacitor power</td><td>Battery and supercapacitors SOC deviation, power imbalance</td><td>HEV</td></tr><tr><td>[40]</td><td>DDPG</td><td>Battery SOC, longitudinal speed, yaw rate</td><td>Rotational speed, torque</td><td>Fuel consumption, electricity consumption, SOC target error</td><td>HEV</td></tr><tr><td>[38]</td><td>Actor-critic</td><td>Battery SOC, Speed, demand power</td><td>Gear position, engine torque</td><td>Fuel consumption, SOC target error, irrational control signals</td><td>HEV</td></tr><tr><td>[50]</td><td>Actor-critic</td><td>Power demand, engine coolant temperature, ambient temperature, velocity, acceleration, battery SOC, traffic and terrain information</td><td>Engine power, drive mode</td><td>Engine fueling rate, battery SOC deviation</td><td>PHEV</td></tr><tr><td>[35]</td><td>Hybrid DDPG and DQN</td><td>Real-time speed, target speed, real angle, road angle, battery SOC, gear position, slope, engine power</td><td>Acceleration, change in steering angle, change in engine power</td><td>Angle difference, speed difference, SOC difference, fuel consumption, engine efficiency</td><td>HEV</td></tr><tr><td>[95]</td><td>Rule-DDPG</td><td>Battery SOH, Battery SOC, demand power</td><td>FC power</td><td>Hydrogen consumption, SOC target error, FC power change rate, operations time</td><td>FCHEV</td></tr><tr><td>[96]</td><td>Adaptive deep Q-learning</td><td>Driver behavior, speed, battery and supercapacitor SOCs, demand power</td><td>Fuel Cell power, battery power</td><td>Battery SOC deviation, hydrogen consumption</td><td>FCHEV</td></tr><tr><td>[49]</td><td>Multi-Agent DDPG</td><td>Velocity, acceleration, battery SOC, ultracapacitor SOC</td><td>Engine torque, front motor torque</td><td>Fuel consumption, battery and ultracapacitor SOCs reference error</td><td>HEV</td></tr><tr><td>[45]</td><td>Visual-based Deep Q-learning</td><td>Distance between the two cars, speed, acceleration, battery SOC, gear ratio</td><td>Change in engine power and gear ratio</td><td>Fuel consumption, SOC and gear ratio deviations, distance error</td><td>HEV</td></tr><tr><td>[48]</td><td>Fuzzy-TD3</td><td>Demand power, vehicle speed, acceleration, battery SOC, ultracapacitor SOC, and power</td><td>Fuel cell power, battery power</td><td>Total hydrogen consumption, fuel cell and battery degradation, battery SOC deviation</td><td>FCHEV</td></tr><tr><td>[37]</td><td>Soft Actor-Critic</td><td>Battery SOC, demand torque, vehicle speed, fuel consumption</td><td>ICE control signal</td><td>Fuel consumption rate, SOC target error</td><td>HEV</td></tr><tr><td>[101]</td><td>DQN</td><td>Required power, battery SOC, battery SOH</td><td>Engine power</td><td>Fuel consumption, SOC target error, battery SOH</td><td>HEV</td></tr><tr><td>[42]</td><td>Multiagent double DQN</td><td>Energy demand, solar energy, battery SOC, buying and selling prices</td><td>Total energy cost, energy profit, and import energy from grid</td><td>Energy efficiency, consumer comfort</td><td>EV</td></tr><tr><td>[93]</td><td>TD3 with safety layer</td><td>Battery SOC, vehicle speed, output power, engine power</td><td>Engine power</td><td>Fuel consumption, SOC target error</td><td>HEV</td></tr><tr><td>[97]</td><td>TD3</td><td>wheel torque, ICE torque, ICE speed, vehicle speed</td><td>ICE torque</td><td>Rate of change of fuel and battery SOC</td><td>HEV</td></tr><tr><td>[46]</td><td>Imitation learning with PPO</td><td>Battery SOC, FC degradation, battery aging, battery output power, FC power, speed, acceleration</td><td>FC power</td><td>Hydrogen consumption, FC degradation, battery aging, battery SOC deviation</td><td>FCHEV</td></tr><tr><td>[98]</td><td>Dual actor-imitator parameterized DQN</td><td>Battery SOC, demand, renewable output, electricity price</td><td>V2G power control</td><td>V2G benefit, battery degradation, and renewable energy consumption rate</td><td>EV</td></tr><tr><td>[99]</td><td>PPO with GAE and LN</td><td>Battery SOC, supercapacitor SOC, load current</td><td>Supercapacitor current</td><td>Operation cost, penalty reward, incentive reward</td><td>BEV</td></tr><tr><td>[100]</td><td>GAIL-based DDPG</td><td>Battery SOC, supercapacitor SOC, demand power</td><td>Supercapacitor current</td><td>Battery capacity loss cost, power loss cost, penalty term</td><td>BEV</td></tr></tbody></table>
<figure><img src="image_23.png" /><figcaption>Fig. 17. Transfer learning concept illustration.</figcaption></figure>

and energy efficiency, with PPO demonstrating rapid learning and significant fuel-saving potential. In contrast, SAC excels in adaptability, making it particularly effective for diverse driving cycles. DDPG has a better convergence time than SAC, meaning it generally learns faster. However, DDPG is less stable and more prone to diverging or getting stuck in local optima. SAC achieves higher final performance and better stability despite taking longer to converge.

Traditional methods like Q-learning and DQN face challenges in continuous control environments, limiting their applicability for HEV EMS. The chart presented in Fig. 18 summarizes the performance scores of each algorithm. The comparison was based on three key performance indicators: Convergence, which measures how quickly and stably an algorithm learns an optimal energy management policy; Effectiveness, which assesses the algorithm's ability to optimize fuel efficiency, SOC deviation, and energy distribution; and Adaptability, which evaluates the algorithm's generalization capability across diverse driving cycles and operational conditions. This comparative analysis may offer insights into the selection of DRL methods for HEV energy management, highlighting the importance of balancing training stability, real-time adaptability, and computational efficiency.

A comprehensive review of DRL-based EMSs can be provided based on another study [115], highlighting key insights into algorithm performance. PPO stands out for its rapid convergence, significantly reducing training time compared to other approaches. Soft Actor-Critic (SAC) excels in achieving the highest cumulative rewards and enhanced fuel efficiency, though it converges more slowly. TD3 offers a more stable learning process than SAC and DDPG, ensuring better adaptability to varying driving cycles. Meanwhile, DQN and its extensions (D2QN, D3QN, Rainbow DQN) demonstrate efficient computation and moderate convergence speeds, making them suitable for real-time applications. Figs. 19 and 20 illustrate the comparison of convergence time

for different DRL algorithms in continuous and discrete action spaces, based on the results shown in [115], highlighting variations in training efficiency across various energy management strategies.

Enhancing the convergence speed of DRL algorithms requires strategic optimizations across various aspects of the training process. One effective technique is hyperparameter tuning, involving the refinement of key parameters such as the learning rate, discount factor, and exploration-exploitation balance. Implementing an adaptive learning rate or decay schedule can dynamically adjust step sizes, ensuring stable convergence and preventing suboptimal local minima. For instance, an adaptive learning rate optimizer based on the Grünwald–Letnikov fractional-order derivative has been proposed to enhance convergence in deep learning models [116]. Similarly, fine-tuning the discount factor (γ) allows the agent to balance short-term and long-term rewards effectively, thereby accelerating learning in complex environments. An adaptive discount factor method has been introduced to adjust γ dynamically during training, leading to improved performance in uncertain environments [117]. In addition, reward shaping and environment design play a crucial role in accelerating convergence. Providing dense rewards and designing a compact state representation can make the learning process more efficient by offering clearer feedback.

Furthermore, employing prioritized experience replay (PER) enables the agent to sample and learn more effectively from high-priority experiences, significantly improving sample efficiency and convergence speed [118]. Intrinsic motivation mechanisms, such as curiosity-driven exploration, further enhance learning by encouraging the agent to explore underrepresented or high-impact states, leading to a more comprehensive understanding of the environment [119].

In the context of HEV energy management systems, improving the convergence speed of DRL algorithms is crucial for enhancing computational effectiveness. Recent studies have explored various strategies to
<figure><img src="image_24.png" /><figcaption>Fig. 18. Comparison of DRL algorithms for HEV energy management based on convergence, effectiveness, and adaptability.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>Fig. 19. Comparison of convergence time for DRL algorithms in continuous action space.</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>Fig. 20. Comparison of convergence time for DRL algorithms in discrete action space. Abbreviations: IQN (Implicit Quantile value Network), D2QN (Double Deep Q-Network), D3QN (Dueling Double Deep Q-Network), FQF (Fully parameterized Quantile Function), C51 (Categorical DQN), QR-DQN (Quantile Regression DQN).</figcaption></figure>

address this challenge. For instance, Lian et al. [120] integrated expert knowledge into DRL-based energy management strategies, demonstrating that such integration not only improves fuel economy but also accelerates convergence efficiency. By incorporating domain-specific insights, the learning process becomes more directed and efficient. In addition, an improved DRL algorithm, based on TD3, has been proposed

in [121], which improves sampling efficiency by introducing dual replay buffers and a rule-based constraint controller. This approach led to a 31.6% improvement in convergence efficiency compared to conventional TD3 algorithms, highlighting the potential of algorithmic modifications to speed up convergence.
# 6. RL EMS research gaps and perspectives

There are some research gaps and challenges in applying RL to HEV energy management. Indeed, most of the studies presented in the literature were conducted in simulations. Although this is an appropriate and reasonable approach to dealing with these problems, RL methods rely on an accurate simulator design and data representing real-world situations. There is also an outstanding discrepancy between simulation and reality, which makes it difficult for the simulated EMS to perform as well as it does in the real world. Since RL is an online learning algorithm, it could be more accurate and effective if it was implemented directly in a physical vehicle to train the energy management system instead of learning in a simulated environment. However, this approach will be risky, time-consuming, and costly. Indeed, the learning period will be very long due to the trial-and-error phase in the real environment. One potential solution is to design a high-precision training simulator to generate the optimal policy and then adjust it adaptively in real-world applications [44].

All RL strategies rely on the quality and quantity of training data. Hence, more in-depth data analysis must be conducted [92] to exploit them in the learning process efficiently. Contemporary vehicles are typically fitted with a variety of sensors. Thus, gathering more information from the vehicle sensors will improve the environment description, accuracy, EMS performance, and fuel economy. However, this will increase the size of the state space and, therefore, the learning time, so there is a trade-off to manage. In addition, we need to improve the quality of the data by adopting appropriate processing methods and avoiding noisy, incomplete, or inaccurate data. We need to ensure that the training data is representative and covers all scenarios that previously occurred as well as those that are currently occurring. Using unrepresentative data in the training process leads to less accurate decisions and faulty predictions.

In addition, with the use of Global Positioning System (GPS) and Geographic Information System (GIS) in vehicles, it is now possible to identify and share information such as road conditions, vehicle status, traffic situation, future speed, energy demand forecast, and other driving circumstances. Using this valuable information will improve the robustness and adaptability of the EMS to changing traffic conditions. Weather conditions, electricity, and fuel costs are important factors in drivers behavior and also in power demand. Adjusting the RL-based EMS based on these factors is an important open question. Therefore, transmitting online information to many vehicles is a promising tool to enhance energy management performance. The establishment of such a smart network is a real challenge because most vehicles must be intelligent and roads must be equipped with high-tech tools. Vehicle-to-everything (V2X) communication system or cloud platform must also be deployed. In this context, vehicles will have to cooperate between them. A central controller must then achieve each vehicle's control objective while considering the mutual influence between all connected vehicles. This challenge can be handled using advanced RL approaches such as asynchronous gradient descent achieving fast and parallel computation [122].

In most RL-based EMSs, the choice of reward signal is both empirical and subjective, and it is not always obvious how to objectively represent the expert's goal [123]. In this context, the RL algorithm cannot ensure that the agent will learn the optimal driving policy under a fixed reward function. Therefore, inverse reinforcement learning (IRL) can be applied to address this issue, and also when control objectives are unknown. The IRL can search for the appropriate reward signal through trial-and-error experiments, and the updated reward function is later reintroduced into the IRL's training.

Another challenge for RL and DRL methods in HEV energy management is handling safety and operational constraints, such as battery SOC limits, power demand satisfaction, and thermal management. Unlike model predictive control (MPC) and other optimization-based

methods that apply “hard constraints” using mathematical formulations, RL/DRL often resolves these constraints by adding penalty terms in the reward function. This “soft constraint” approach relies on an agent learning to avoid constraint violations by trial and error, which does not guarantee exact adherence to safety constraints, particularly in the early training phases. During exploration, for instance, the agent might temporarily exceed SOC or power limits, leading to suboptimal or even unsafe operation. This raises questions about the robustness and reliability of RL algorithms in safety-critical applications. To address this problem, few studies have proposed hybrid approaches in which RL/DRL is combined with rule-based strategies, MPC, or constrained policy optimization to handle hard constraints. Nevertheless, further research should be conducted to propose more effective methods for incorporating safety constraints in RL/DRL-based EMSs while ensuring their flexibility and learning capabilities are not compromised.

The efficient training and online adaption of the learning algorithms have also been a major challenge in RL/DRL-based EMS. Training RL/DRL models often requires extensive computational resources and large datasets, which is both time-consuming and costly. In addition, the actual training process is typically conducted offline using pre-recorded driving cycles that may not fully capture the dynamic and stochastic characteristics of real-world driving scenarios. This requirement presents a challenge in EMS applications because communication latencies and computational overheads can limit the processing and acquisition of data in real-time. Furthermore, renewable energy is non-stationary in nature and the load demand is dynamic, which usually requires online adaptation of the control policy. This difference between training and real-world conditions can result in suboptimal performance when the RL/DRL agent is deployed online. However, RL/DRL methods can effectively reduce the exploration phase and improve sample efficiency by pre-training the model using expert demonstrations or optimal trajectories obtained via offline optimization methods (e.g., dynamic programming or MILP). Other approaches, including transfer learning (TL) and online learning, with the agent updating the policy online using real-time data, have also been investigated by researchers [89,105]. This reduces the time and data required for training on new tasks or environments, as the agent can build on previous experience through transfer learning. In contrast, online learning allows the agent to adjust to dynamic environments immediately, enabling better robustness and generalizability. However, these approaches come with their own challenges, such as the need for high-quality real-time data and the risk of overfitting to transient conditions. Approaches towards efficient training and online updates are still under investigation, with promising directions such as meta-learning, ensemble methods, and intelligent exploration techniques that could potentially enhance the adaptability and efficiency of RL/DRL-based EMS.

DRL-based EMSs for HEVs rely on high-quality driving data. However, the presence of abnormal data caused by sensor malfunctions, communication errors, or adversarial attacks can significantly degrade training outcomes. Data inconsistencies introduce biases that lead to suboptimal decision-making, reduced energy efficiency, and potential safety risks. Several studies have shown that incomplete or incorrect training data can disrupt the learning process, resulting in poor generalization under different driving conditions. Another issue arises from model collapse due to corrupted or low-quality driving data. In model-free RL methods like Q-learning and DDPG, training on anomalous or highly biased data can lead to policies that fail to generalize beyond specific driving conditions. This is particularly problematic in HEVs, where diverse environmental and traffic factors must be considered for optimal energy management. Furthermore, abnormal data contribute to catastrophic interference [124], where the DRL model forgets previously learned energy optimization strategies when exposed to inconsistent or contradictory training samples. This issue is especially relevant in continuous online learning settings, where HEV EMS must adapt to real-time driving conditions without losing past knowledge.
To mitigate these issues, data validation, anomaly detection, and corrective machine unlearning techniques should be integrated into HEV RL-based EMS frameworks [125]. In addition, leveraging hybrid model-based DRL approaches that use predictive models to filter out abnormal data before training ensures reliable and robust decision-making.

## 7. Conclusion

Energy management systems for hybrid electric vehicles are important for their daily optimal economic and reliable operation. Reinforcement learning helps improve the energy management system by using physical models and real-time data. This paper provided a comprehensive comparative analysis of different reinforcement learning methods used for the energy management of hybrid electric vehicles. A comparison of the advantages and limitations of these reinforcement learning algorithms was also provided. The objectives, key parameters, and constraints for energy management strategies for hybrid electric vehicles were also comprehensively discussed. States, actions, and reward objectives were also provided for reinforcement learning-based energy management systems. It can be concluded from this extensive review that reinforcement learning-based energy management systems achieve a considerable improvement in the energy economy of hybrid electric vehicles. The energy optimization compared to conventional energy management systems differs from one application to another according to the specific driving conditions and energy savings were achieved between 2.18% to 26%. It is also analyzed that reinforcement learning generates near-optimal policies compared to the benchmark optimal strategy typically represented by dynamic programming. Besides, neural network integration has shown improved effectiveness over standard reinforcement learning methods. Learning convergence time is significantly reduced relative to standard offline optimization methods, especially when using deep transfer learning. However, most of the investigated reinforcement learning-based EMSs are limited to simulation setups. Thus, accurate and powerful simulators are needed, and high-quality data representing real conditions should be obtained during the pre-training stage. Online information sharing between neighboring hybrid electric vehicles can also help to improve energy management performance.

## CRediT authorship contribution statement

**Mohamed Nadir Boukoberine:** Writing – original draft, Visualization, Validation, Methodology, Investigation, Formal analysis.
**Muhammad Fahad Zia:** Writing – review & editing, Visualization, Validation, Methodology, Formal analysis.
**Tarek Berghout:** Writing – review & editing, Visualization.
**Mohamed Benbouzid:** Writing – review & editing, Visualization, Validation, Supervision, Methodology, Investigation, Formal analysis.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

No data was used for the research described in the article.

## References

[1] Orzechowski A, Lugosch L, Shu H, Yang R, Li W, Meyer BH. A data-driven framework for medium-term electric vehicle charging demand forecasting. Energy AI 2023;14:100267.

[2] Precedence Research. Hybrid vehicle market size, growth, trends 2024 to 2034. 2025, URL https://www.precedenceresearch.com/hybrid-vehicle-market. [Accessed: 06 February 2025].

[3] Zhao M, Fang Y, Dai D. Forecast of the evolution trend of total vehicle sales and power structure of China under different scenarios. Sustainability 2023;15(5):3985.

[4] Zhang F, Hu X, Langari R, Cao D. Energy management strategies of connected HEVs and PHEVs: Recent progress and outlook. Prog Energy Combust Sci 2019;73:235-56.

[5] Sabri M, Danapalasingam KA, Rahmat MF. A review on hybrid electric vehicles architecture and energy management strategies. Renew Sustain Energy Rev 2016;53:1433-42.

[6] Boukoberine MN, Zhou Z, Benbouzid M. A critical review on unmanned aerial vehicles power supply and energy management: Solutions, strategies, and prospects. Appl Energy 2019;255:113823.

[7] Saiteja P, Ashok B. Critical review on structural architecture, energy control strategies and development process towards optimal energy management in hybrid vehicles. Renew Sustain Energy Rev 2022;157:112038.

[8] Boukoberine MN, Zhou Z, Benbouzid M, Donateo T. Frequency separation-based power management strategy for a fuel cell-powered drone. In: 2020 2nd international conference on smart power internet energy systems. 2020, p. 209-14.

[9] Mohd Sabri MF, Danapalasingam KA, Rahmat MF. Improved fuel economy of through-the-road hybrid electric vehicle with fuzzy logic-based energy management strategy. Int J Fuzzy Syst 2018;20(8):2677-92.

[10] Li X, Wang Y, Yang D, Chen Z. Adaptive energy management strategy for fuel cell/battery hybrid vehicles using Pontryagin's Minimal Principle. J Power Sources 2019;440:227105.

[11] Boukoberine MN, Zia MF, Benbouzid M, Zhou Z, Donateo T. Hybrid fuel cell powered drones energy management strategy improvement and hydrogen saving using real flight test data. Energy Convers Manage 2021;236:113987.

[12] Rodriguez R, Trovão JPF, Solano J. Fuzzy logic-model predictive control energy management strategy for a dual-mode locomotive. Energy Convers Manage 2022;253:115111.

[13] Boukoberine MN, Donateo T, Benbouzid M. Optimized energy management strategy for hybrid fuel cell powered drones in persistent missions using real flight test data. IEEE Trans Energy Convers 2022;37(3):2080-91.

[14] Chen Z, Xiong R, Cao J. Particle swarm optimization-based optimal power management of plug-in hybrid electric vehicles considering uncertain driving conditions. Energy 2016;96:197-208.

[15] Gao Q, Lei T, Deng F, Min Z, Yao W, Zhang X. A deep reinforcement learning based energy management strategy for fuel-cell electric UAV. In: 2022 international conference on power energy systems and applications. 2022, p. 524-30.

[16] Mason K, Grijalva S. A review of reinforcement learning for autonomous building energy management. Comput Electr Eng 2019;78:300-12.

[17] Sutton RS, Barto AG. Reinforcement learning: An introduction. MIT Press; 2018.

[18] Wu J, He H, Peng J, Li Y, Li Z. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Appl Energy 2018;222:799-811.

[19] Han X, He H, Wu J, Peng J, Li Y. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle. Appl Energy 2019;254:113708.

[20] Hu X, Liu T, Qi X, Barth M. Reinforcement learning for hybrid and plug-in hybrid electric vehicle energy management: Recent advances and prospects. IEEE Ind Electron Mag 2019;13(3):16-25.

[21] Li Q, Meng X, Gao F, Zhang G, Chen W, Rajashekara K. Reinforcement learning energy management for fuel cell hybrid systems: A review. IEEE Ind Electron Mag 2022;17(4):45-54.

[22] Narayanan AV, Marion LGR, Gopakumar P. Reinforcement learning based energy management system for hybrid electric vehicles. In: 2023 3rd international conference on advances in computing, communication, embedded and secure systems. 2023, p. 202-8.

[23] Ganesh AH, Xu B. A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution. Renew Sustain Energy Rev 2022;154:111833.

[24] He H, Meng X, Wang Y, Khajepour A, An X, Wang R, et al. Deep reinforcement learning based energy management strategies for electrified vehicles: Recent advances and perspectives. Renew Sustain Energy Rev 2024;192:114248.

[25] Tang X, Chen J, Qin Y, Liu T, Yang K, Khajepour A, et al. Reinforcement learning-based energy management for hybrid power systems: state-of-the-art survey, review, and perspectives. Chin J Mech Eng 2024;37(1):43.

[26] Shen H, Zhang Y, Mao J, Yan Z, Wu L. Energy management of hybrid UAV based on reinforcement learning. Electronics 2021;10(16):1929.
[27] Meng X, Li Q, Zhang G, Wang X, Chen W. Double Q-learning-based energy management strategy for overall energy consumption optimization of fuel cell/battery vehicle. In: 2021 IEEE transportation electrification conference & expo. IEEE, 2021, p. 1-6.

[28] Li Q, Meng X, Gao F, Zhang G, Chen W. Approximate cost-optimal energy management of hydrogen electric multiple unit trains using double Q-learning algorithm. IEEE Trans Ind Electron 2022;69(9):9099-110.

[29] Yang N, Han L, Liu R, Wei Z, Liu H, Xiang C. Multiobjective intelligent energy management for hybrid electric vehicles based on multiagent reinforcement learning. IEEE Trans Transp Electrification 2023;9(3):4294-305.

[30] Sun H, Fu Z, Tao F, Zhu L, Si P. Data-driven reinforcement-learning-based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles. J Power Sources 2020;455:227964.

[31] Ye Y, Wang H, Xu B, Zhang J. An imitation learning-based energy management strategy for electric vehicles considering battery aging. Energy 2023;283:128537.

[32] Hu B, Liu B, Zhang S. A data-driven reinforcement learning based energy management strategy via bridging offline initialization and online fine-tuning for a hybrid electric vehicle. IEEE Trans Ind Electron 2024.

[33] Li K, Chen H, Wu Y, Zhao J, Ding S, Gao J. A model-free combined energy and thermal management strategy for HEVs based on reinforcement-learning under low-temperature. IEEE Trans Intell Veh 2024.

[34] Qi C, Song C, Wang D, Xiao F, Jin L, Song S. Action advising and energy management strategy optimization of hybrid electric vehicle agent based on uncertainty analysis. IEEE Trans Transp Electrification 2023.

[35] Chen J, Li S, Yang K, Wei C, Tang X. Deep reinforcement learning-based integrated control of hybrid electric vehicles driven by lane-level high-definition map. IEEE Trans Transp Electrification 2023;10(1):1642-55.

[36] Li Y, He H, Peng J, Wang H. Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information. IEEE Trans Veh Technol 2019;68(8):7416-30.

[37] Wang J, Du C, Yan F, Duan X, Hua M, Xu H, et al. Energy management of a plug-in hybrid electric vehicle using Bayesian optimization and soft actor-critic algorithm. IEEE Trans Transp Electrification 2024.

[38] Liu ZE, Li Y, Zhou Q, Li Y, Shuai B, Xu H, et al. Deep reinforcement learning based energy management for heavy duty HEV considering discrete-continuous hybrid action space. IEEE Trans Transp Electrification 2024.

[39] Qi X, Wu G, Boriboonsomsin K, Barth MJ, Gonder J. Data-driven reinforcement learning-based real-time energy management system for plug-in hybrid electric vehicles. Transp Res Rec 2016;2572(1):1-8.

[40] Han R, Lian R, He H, Han X. Continuous reinforcement learning-based energy management strategy for hybrid electric-tracked vehicles. IEEE J Emerg Sel Top Power Electron 2021;11(1):19-31.

[41] Jia C, Liu W, He H, Chau K. Deep reinforcement learning-based energy management strategy for fuel cell buses integrating future space information and cabin comfort control. Energy Convers Manage 2024;321:119032.

[42] Yavuz M, Kivanç ÖC. Optimization of a cluster-based energy management system using deep reinforcement learning without affecting prosumer comfort: V2X technologies and peer-to-peer energy trading. IEEE Access 2024.

[43] Wu P, Partridge J, Bucknall R. Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships. Appl Energy 2020;275:115258.

[44] Wu Y, Tan H, Peng J, Zhang H, He H. Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus. Appl Energy 2019;247:454-66.

[45] Tang X, Chen J, Yang K, Toyoda M, Liu T, Hu X. Visual detection and deep reinforcement learning-based car following and energy management for hybrid electric vehicles. IEEE Trans Transp Electrification 2022;8(2):2501-15.

[46] Peng J, Ren T, Chen Z, Chen W, Wu C, Ma C. Efficient training for energy management in fuel cell hybrid electric vehicles: An imitation learning-embedded deep reinforcement learning framework. J Clean Prod 2024;447:141360.

[47] Zhang J, Tao J, Hu Y, Ma L. An energy management strategy based on DDPG with improved exploration for battery/supercapacitor hybrid electric vehicle. IEEE Trans Intell Transp Syst 2023.

[48] Wang H, Wang X, Fu Z. Energy management strategy for optimal charge depletion of plug-in FCHEV based on multi-constraints deep reinforcement learning. IEEE Trans Transp Electrification 2024.

[49] Gan J, Li S, Lin X, Tang X. Multi-agent deep reinforcement learning-based multi-objective cooperative control strategy for hybrid electric vehicles. IEEE Trans Veh Technol 2024.

[50] Zhang H, Chen B, Lei N, Li B, Li R, Wang Z. Integrated thermal and energy management of connected hybrid electric vehicles using deep reinforcement learning. IEEE Trans Transp Electrification 2023.

[51] Liu T, Wang B, Yang C. Online Markov chain-based energy management for a hybrid tracked vehicle with speedy Q-learning. Energy 2018;160:544-55.

[52] Kolodziejczyk W, Zoltowska I, Cichosz P. Real-time energy purchase optimization for a storage-integrated photovoltaic system by deep reinforcement learning. Control Eng Pract 2021;106:104598.

[53] Xu B, Rathod D, Zhang D, Yebi A, Zhang X, Li X, et al. Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle. Appl Energy 2020;259:114200.

[54] Li Y, He H, Peng J, Wang H. Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information. IEEE Trans Veh Technol 2019;68(8):7416-30.

[55] Hu B, Xiao Y, Zhang S, Liu B. A data-driven solution for energy management strategy of hybrid electric vehicles based on uncertainty-aware model-based offline reinforcement learning. IEEE Trans Ind Informatics 2022;19(6):7709-19.

[56] Liu Y, Wu Y, Wang X, Li L, Zhang Y, Chen Z. Energy management for hybrid electric vehicles based on imitation reinforcement learning. Energy 2023;263:125890.

[57] Yang D, Wang S, Liao Y, Pan R. An online energy management strategy for fuel cell vehicles based on fuzzy Q-learning and road condition recognition. IEEE Trans Intell Transp Syst 2024;25(9):12120-30.

[58] Lin Y, Chu L, Hu J, Hou Z, Li J, Jiang J, et al. Progress and summary of reinforcement learning on energy management of MPS-EV. Heliyon 2024;10(1).

[59] Lu L, Zhao H, Xv F, Luo Y, Chen J, Ding X. GA-LSTM speed prediction-based DDQN energy management for extended-range vehicles. Energy AI 2024;17:100367.

[60] Guo L, Li Z, Outib R, Gao F. Function approximation reinforcement learning of energy management with the fuzzy REINFORCE for fuel cell hybrid electric vehicles. Energy AI 2023;13:100246.

[61] Shen S, Gao S, Liu Y, Zhang Y, Shen J, Chen Z, et al. Real-time energy management for plug-in hybrid electric vehicles via incorporating double-delay Q-learning and model prediction control. IEEE Access 2022;10:131076-89.

[62] Xiong R, Cao J, Yu Q. Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle. Appl Energy 2018;211:538-48.

[63] Shuai B, Hua M, Li Y, Shuai S, Xu H, Zhou Q. Optimal energy management of plug-in hybrid electric vehicles through ensemble reinforcement learning with exploration-to-exploitation ratio control. IEEE Trans Intell Veh 2024.

[64] Li X, Zhou Z, Wei C, Gao X, Zhang Y. Multi-objective optimization of hybrid electric vehicles energy management using multi-agent deep reinforcement learning framework. Energy AI 2025;100491.

[65] Ahmadian S, Tahmasbi M, Abedi R. Q-learning based deep reinforcement learning for energy management of series-parallel hybrid vehicles with balanced fuel consumption and battery life. Energy AI 2023;11:100217.

[66] Watkins CJ, Dayan P. Q-learning. Mach Learn 1992;8(3):279-92.

[67] Jaakkola T, Jordan M, Singh S. Convergence of stochastic iterative dynamic programming algorithms. Adv Neural Inf Process Syst 1993;6.

[68] Even-Dar E, Mansour Y, Bartlett P. Learning rates for Q-learning. J Mach Learn Res 2003;5(1).

[69] Ghavamzadeh M, Kappen H, Azar M, Munos R. Speedy Q-learning. Adv Neural Inf Process Syst 2011;24.

[70] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning. In: Proceedings of the AAAI conference on artificial intelligence, vol. 30, 2016.

[71] Hasselt H. Double Q-learning. Adv Neural Inf Process Syst 2010;23.

[72] Liu T. Reinforcement learning-enabled intelligent energy management for hybrid electric vehicles. Springer Nature; 2022.

[73] Liu T, Zou Y, Liu D, Sun F. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle. IEEE Trans Ind Electron 2015;62(12):7837-46.

[74] Bai M, Yang W, Song D, Kosuda M, Szabo S, Lipovsky P, et al. Research on energy management of hybrid unmanned aerial vehicles to improve energy-saving and emission reduction performance. Int J Environ Res Public Health 2020;17(8):2917.

[75] Lin X, Zhou B, Xia Y. Online recursive power management strategy based on the reinforcement learning algorithm with cosine similarity and a forgetting factor. IEEE Trans Ind Electron 2021;68(6):5013-23.

[76] Hu J, Lin Y, Li J, Hou Z, Chu L, Zhao D, et al. Performance analysis of AI-based energy management in electric vehicles: A case study on classic reinforcement learning. Energy Convers Manage 2024;300:117964.

[77] Xu B, Hu X, Tang X, Lin X, Li H, Rathod D, et al. Ensemble reinforcement learning-based supervisory control of hybrid electric vehicle for fuel economy improvement. IEEE Trans Transp Electrification 2020;6(2):717-27.

[78] Huang Z, Wang J, Fan X, Yue R, Xiang C, Gao S. Economical electric vehicle charging scheduling via deep imitation learning. IEEE Trans Intell Transp Syst 2024;25(11):18196-210.

[79] Hussein A, Gaber MM, Elyan E, Jayne C. Imitation learning: A survey of learning methods. ACM Comput Surv 2017;50(2):1-35.

[80] Arwa EO, Folly KA. Reinforcement learning techniques for optimal power control in grid-connected microgrids: A comprehensive review. IEEE Access 2020;8:208992-9007.

[81] Hu Y, Li W, Xu K, Zahid T, Qin F, Li C. Energy management strategy for a hybrid electric vehicle based on deep reinforcement learning. Appl Sci 2018;8(2):187.

[82] Li Y, He H, Khajepour A, Wang H, Peng J. Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information. Appl Energy 2019;255:113762.

[83] Wu Z, Ma B. Research on HEV energy management strategy based on improved deep reinforcement learning. J Ind Manag Optim 2023;19(12):8451-68.
[84] Li Y, He H. Deep reinforcement learning-based energy management for hybrid electric vehicles. Morgan & Claypool Publishers; 2022.

[85] Li W, Cui H, Nemeth T, Jansen J, Ünlübayir C, Wei Z, et al. Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles. J Energy Storage 2021;36:102355.

[86] Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res 2012;13(2).

[87] Kong H, Yan J, Wang H, Fan L. Energy management strategy for electric vehicles based on deep Q-learning using Bayesian optimization. Neural Comput Appl 2020;32(18):14431-45.

[88] Chowdhury AA, Hossen MA, Azam MA, Rahman MH. DeepQGHO: Quantized greedy hyperparameter optimization in deep neural networks for on-the-fly learning. IEEE Access 2022;10:6407-16.

[89] Zou R, Fan L, Dong Y, Zheng S, Hu C. DQL energy management: An online-updated algorithm and its application in fix-line hybrid electric vehicle. Energy 2021;225:120174.

[90] Zhang X, Wang X, Yuan P, Tian H, Shu G. Optimizing hybrid electric vehicle coupling organic rankine cycle energy management strategy via deep reinforcement learning. Energy AI 2024;17:100392.

[91] Wang X, Wang R, Shu G, Tian H, Zhang X. Energy management strategy for hybrid electric vehicle integrated with waste heat recovery system based on deep reinforcement learning. Sci China Technol Sci 2022;65(3):713-25.

[92] Tang X, Chen J, Liu T, Qin Y, Cao D. Distributed deep reinforcement learning-based energy and emission management strategy for hybrid electric vehicles. IEEE Trans Veh Technol 2021;70(10):9922-34.

[93] Biswas A, Acquarone M, Wang H, Miretti F, Misul DA, Emadi A. Safe reinforcement learning for energy management of electrified vehicle with novel physics-informed exploration strategy. IEEE Trans Electrification 2024.

[94] Zhou J, Xue S, Xue Y, Liao Y, Liu J, Zhao W. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning. Energy 2021;224:120118.

[95] Tang X, Shi L, Zhang Y, Li B, Xu S, Song Z. Degradation adaptive energy management strategy for FCHEV based on the rule-DDPG method: tailored to the current SOH of the powertrain. IEEE Trans Transp Electrification 2024.

[96] Sun H, Tao F, Fu Z, Gao A, Jiao L. Driving-behavior-aware optimal energy management strategy for multi-source fuel cell hybrid electric vehicles based on adaptive soft deep-reinforcement learning. IEEE Trans Intell Transp Syst 2023;24(4):4127-46.

[97] Hu Y, Xu H, Jiang Z, Zheng X, Zhang J, Fan W, et al. Supplementary learning control for energy management strategy of hybrid electric vehicles at scale. IEEE Trans Veh Technol 2023;72(6):7290-303.

[98] Li S, Zhao AP, Gu C, Bu S, Chung E, Tian Z, et al. Interpretable deep reinforcement learning with imitative expert experience for smart charging of electric vehicles. IEEE Trans Power Syst 2025;40(2):1228-40.

[99] Li F, Gao Y, Wu Y, Xia Y, Wang C, Hu J, et al. Incentive learning-based energy management for hybrid energy storage system in electric vehicles. Energy Convers Manage 2023;293:117480.

[100] Liu W, Yao P, Wu Y, Duan L, Li H, Peng J. Imitation reinforcement learning energy management for electric vehicles with hybrid energy storage system. Appl Energy 2025;378:124832.

[101] Yang N, Han L, Zhou X, Liu R, Liu H, Xiang C. Online-learning adaptive energy management for hybrid electric vehicles in various driving scenarios based on dynamic framework. IEEE Trans Transp Electrification 2023.

[102] Xiong R, Duan Y, Cao J, Yu Q. Battery and ultracapacitor in-the-loop approach to validate a real-time power management method for an all-climate electric vehicle. Appl Energy 2018;217:153-65.

[103] Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C. A survey on deep transfer learning. In: International conference on artificial neural networks. Springer; 2018, p. 270-9.

[104] Weiss K, Khoshgoftaar TM, Wang D. A survey of transfer learning. J Big Data 2016;3(1):1-40.

[105] Lian R, Tan H, Peng J, Li Q, Wu Y. Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management. IEEE Trans Veh Technol 2020;69(8):8367-80.

[106] Taylor ME, Stone P. Transfer learning for reinforcement learning domains: A survey. J Mach Learn Res 2009;10(7):1633-85.

[107] Guo X, Liu T, Tang B, Tang X, Zhang J, Tan W, et al. Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle. IEEE Access 2020;8:165837-48.

[108] Xu J, Li Z, Gao L, Ma J, Liu Q, Zhao Y. A comparative study of deep reinforcement learning-based transferable energy management strategies for hybrid electric vehicles. 2022, http://dx.doi.org/10.48550/arXiv.2202.11514, arXiv preprint arXiv:2202.11514.

[109] Hou S, Gao S, Xia W, Salazar Duque EM, Palensky P, Vergara PP. RL-ADN: A high-performance deep reinforcement learning environment for optimal energy storage systems dispatch in active distribution networks. Energy AI 2025;19:100457.

[110] Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Beabeel P. Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ international conference on intelligent robots and systems. IEEE; 2017, p. 23-30.

[111] Li Y, Zhao SZ, Xu C, Tang C, Li C, Ding M, et al. Pre-training on synthetic driving data for trajectory prediction. In: 2024 IEEE/RSJ international conference on intelligent robots and systems. IEEE; 2024, p. 5910-7.

[112] Wang R, Shi X, Su Y, Song T. A predictive energy management strategy for plug-in hybrid electric vehicles using real-time traffic based reference SOC planning. Proc Inst Mech Eng Part D: J Automob Eng 2024;09544070241239996.

[113] Wang JX, Kurth-Nelson Z, Tirumala D, Soyer H, Leibo JZ, Munos R, et al. Learning to reinforcement learn. 2016, arXiv preprint arXiv:1611.05763.

[114] Xiong L, Tang Y, Liu C, Mao S, Meng K, Dong Z, et al. Meta-reinforcement learning-based transferable scheduling strategy for energy management. IEEE Trans Circuits Syst I Regul Pap 2023;70(4):1685-95.

[115] Wang H, Ye Y, Zhang J, Xu B. A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle. Energy 2023;266:126497.

[116] Zhang X, Wang Y, Li J. An adaptive learning rate optimizer based on Grünwald-Letnikov fractional-order derivative. Neural Process Lett 2024;58:1-20. http://dx.doi.org/10.1007/s11063-024-11571-7.

[117] Chen L, Ma R, Liu H. An adaptive discount factor method for reinforcement learning in uncertain environments. Sensors 2022;22(19):7266. http://dx.doi.org/10.3390/s22197266.

[118] Schaul T, Quan J, Antonoglou I, Silver D. Prioritized experience replay. 2015, arXiv preprint arXiv:1511.05952.

[119] Pathak D, Agrawal P, Efros AA, Darrell T. Curiosity-driven exploration by self-supervised prediction. In: International conference on machine learning. PMLR; 2017, p. 2778-87.

[120] Lian R, Peng J, Wu Y, Tan H, He H, Wu J. Deep reinforcement learning based energy management of hybrid electric vehicles with expert knowledge. Energy Proc 2019;3:1-10.

[121] Wu J, Ma T. A dual replay buffer-based TD3 algorithm for HEV energy management optimization. AIMS J Ind Manag Optim 2023;19:1456-78. http://dx.doi.org/10.3934/jimo.2023046.

[122] Liu T. Reinforcement learning-enabled intelligent energy management for hybrid electric vehicles. Synth Lect Adv Automot Technol 2019;3(5):1-99.

[123] Lv H, Qi C, Song C, Song S, Zhang R, Xiao F. Energy management of hybrid electric vehicles based on inverse reinforcement learning. Energy Rep 2022;8:5215-24.

[124] Li Z, Tang J, Zhao H, Chen C, Xie S. Dictionary learning-structured reinforcement learning with adaptive-sparsity regularizer. IEEE Trans Aerosp Electron Syst 2023;60(2):1753-69.

[125] Xu J, Wu Z, Wang C, Jia X. Machine unlearning: Solutions and challenges. IEEE Trans Emerg Top Comput Intell 2024.