# Energy Management in Hybrid Energy Storage Systems for Electric Vehicles: A Reinforcement Learning Approach with Python-Simulink Integration

Parisa Ranjbaran, Alireza Bakhshai, Praveen Jain

► To cite this version:

Parisa Ranjbaran, Alireza Bakhshai, Praveen Jain. Energy Management in Hybrid Energy Storage Systems for Electric Vehicles: A Reinforcement Learning Approach with Python-Simulink Integration. The 26th European Conference on Power Electronics and Applications, GDR SEEDS France & EPE Association, Mar 2025, Paris, France. 10.34746/epe2025-0342. hal-05067916

HAL Id: hal-05067916

https://utc.hal.science/hal-05067916v1

Submitted on 14 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Energy Management in Hybrid Energy Storage Systems for Electric Vehicles: A Reinforcement Learning Approach with Python-Simulink Integration

Parisa Ranjbaran, Alireza Bakhshai, and Praveen Jain

Energy and Power Electronics Research (ePower) Group
Department of Electrical and Computer Engineering, Queen's University
Kingston, Canada

parisa.ranjbaran@queensu.ca, alireza.bakhshai@queensu.ca, praveen.jain@queensu.ca

## Keywords

«Energy Management System (EMS)», «Reinforcement Learning», «Electric Vehicle (EV)», «Energy storage».

## Abstract

A Reinforcement Learning (RL) algorithm is employed for the energy management of a Hybrid Energy Storage System (HESS) in All Electric Vehicles (AEVs), focusing on enhancing battery life and vehicle mileage. Simulink_gym is used as the interface between Python and Simulink environment to enable seamless communication for efficient simulation and training. The results indicate that the RL-based Energy Management Strategy (EMS) significantly outperforms conventional rule-based strategies by achieving superior power sharing in HESS, leading to enhanced energy efficiency and more balanced utilization of storage components.

## Introduction

Electric vehicles (EVs) have emerged as a promising solution to address global warming and fuel resource exhaustion. EVs offer a viable alternative to the issues of environmental pollution posed by conventional Internal Combustion Engine (ICE) powered vehicles, due to their potential to achieve reduced emissions and conserve energy. EVs are broadly classified into three main types: All Electric Vehicles (AEVs), which rely solely on battery packs and electric motors for propulsion; Plug-in Hybrid Electric Vehicles (PHEVs), which combine a conventional gasoline or diesel engine with an electric motor and a rechargeable battery that can be directly plugged into the grid to be charged; and Hybrid Electric Vehicles (HEVs), which predominantly use the gasoline engine for

propulsion but employ the electric motor to improve efficiency, particularly during low-speed operations [1].

According to international protocols and agreements targeting zero emissions, AEVs offer distinct advantages over other types of electric vehicles [2]. Unlike HEVs and PHEVs, AEVs operate exclusively on electricity, ensuring they produce no tailpipe emissions. This characteristic positions AEVs at the forefront of efforts to reduce urban air pollution and mitigate the impacts of climate change. Moreover, AEVs benefit from simpler mechanical designs compared to their hybrid counterparts, which often results in reduced maintenance costs and longer vehicle lifespans. Given the rapid expansion of renewable energy sources, the environmental footprint of AEVs continues to diminish as the electricity grid becomes cleaner [3]. Additionally, the infrastructure for AEVs is expanding, further facilitating the use of these vehicles and enhancing their accessibility and convenience for consumers [4]. Thus, from both an environmental and economic standpoint, AEVs are increasingly recognized as the optimal choice for sustainable transportation in alignment with global decarbonization goals [5].

The degradation of batteries in AEVs is a pressing concern, especially given that the battery constitutes a significant portion of the vehicle's cost. Under frequent and high-rate charging and discharging conditions, this degradation becomes even more severe, leading to a shortened battery lifespan. This not only affects the overall economy but also hinders the wider adoption of AEVs. To address these challenges, the integration of supercapacitors (SCs) as supplementary power sources presents a promising solution. SCs, also known as ultracapacitors, possess rapid charge-discharge capabilities and high-power density, making them ideal for absorbing and delivering short bursts of
energy. By combining batteries and SCs in a hybrid system, the instantaneous power demands can be shifted towards the SCs, thereby reducing the stress and frequent high-rate charge-discharge cycles on the batteries. This approach not only prolongs the battery's lifespan but also improves vehicular performance and efficiency.

The effective functioning of a HESS necessitates the deployment of an EMS. This EMS is pivotal in AEVs, dictating the amount of energy production and specifying how energy is stored and expended across various powertrain components. The inclusion of an EMS within a HESS ensures that the energy storage is harnessed efficiently, thereby ensuring the optimal operation of the system. This is achieved by balancing both technical specifications and economic factors. By aligning with the instantaneous power demands of a vehicle, the EMS not only enhances the performance of the system but also extends the HESS's lifespan, broadens the driving range, and reduces operational expenses. The basic characteristics of an EMS encompass real-time implementation speed, adaptivity, robustness, and accuracy.

Generally, energy management strategies are classified into three groups, rule-based strategies, optimization-based strategies, and learning-based strategies. The rule-based strategy is the earliest method of energy management in EVs that works by relying on a certain set of predefined rules. These rules usually consider the current state of the HESS and driving parameters such as the motor's power demand or engine speed. Rule-based strategies are simple but the optimal results are rarely achieved [6]. In addition, optimization-based EMSs often suffer from high computational complexity, especially for real-time applications, as they require solving complex optimization problems at each decision step. Learning-based strategy represents a shift from traditional rule-based or optimization-driven approaches to utilizing the power of machine learning and artificial intelligence. These systems adapt and refine their strategies over time by processing vast amounts of data and learning from past decisions, environmental factors, and system responses. The core advantage of learning-based methods is their ability to predict and adapt to dynamic situations, which results in continuously improving their performance [7]. Among the machine learning algorithms, RL has gained a lot of attention in this field due to its capability to learn optimal control strategies directly from the interaction with the environment [8]. Unlike supervised learning

algorithms, which rely on predefined datasets, RL algorithms dynamically adapt to the changes in the conditions such as changing the driving pattern or power demand, making them well-suited for complex, real-time decision-making. RL algorithms optimize the performance of the system by considering the future reward [9], [10]. It can effectively balance the energy allocation from the battery and SC to maximize energy efficiency, enhance the battery life, increase the mileage of the vehicle, and reduce operational costs [11].

In this regard, many papers have investigated the application of RL algorithms in the energy management of HESS. In order to implement a multi-objective optimization, a multi-agent RL is introduced to address the energy management for a HEV composed of an Engine-Generator Set and a HESS [12]. In another paper, a joint eco-routing and EMS based on the RL algorithms is introduced [13]. A Q-learning algorithm is employed to determine the optimal eco-route and a Soft Actor-Critic (SAC) algorithm is used to perform the energy management. An incentive learning-based EMS is introduced in [14] where a Proximal Policy Optimization (PPO) algorithm is utilized for the energy management of a HESS in an AEV. A semi-supervised support vector machine is used to identify the driving style and enhance the performance of the RL-based energy management [15]. In another article, an imitation learning method is introduced to shorten the training time of the Q-learning in the energy management of a HESS, which could reduce the computational time by 53.28% in comparison with the conventional Q-learning-based EMS [16]. An improved Deep Q-Learning (DQN) method is introduced in [17] in which the reward function is defined using the Equivalent Consumption Minimum Strategy theory of multi-objective optimization to control a fuel cell/battery/SC HESS. In [18] a learning guidance mechanism is used to enhance the learning process of a DDPG algorithm. An adaptive fuzzy filter is also employed to decrease the computational burden of DDPG. A Twin-Delayed Deep Deterministic Policy Gradient (TD3) is used in [19] to design an energy management framework for a fuel cell/battery/SC HESS. A comparison between the DQN-based, DDPG-based, and SAC-based EMSs showed that the SAC-based EMS can reduce energy loss by 8.75% and 6.09% respectively compared to DQN-based and DDPG-based EMSs [11]. A
combined rule-based is presented in [20] for controlling the power-sharing between a lithium battery and an SC. In [21] an RL-based real-time EMS for a HESS in plug-in hybrid electric vehicles is proposed, utilizing a Q-learning algorithm to optimize power distribution between batteries and SCs. It introduces an adaptive approach that recalculates power transition probabilities online, based on actual driving conditions.

In this paper, a Proximal Policy Optimization (PPO) algorithm is employed for energy management in a HESS within an AEV. To this end, the PPO agent is implemented in Python, to make use of its advanced machine-learning libraries, while the HESS model is modeled in MATLAB/Simulink to utilize its robust modeling capabilities. This integration is further distinguished by employing the 'Simulink_gym' wrapper, which facilitates dynamic TCP/IP-based communication between the simulation environment and the RL agent, a capability not explored in previous research studies.

## System description

The HESS serves as a voltage source, supplying DC power to the traction subsystem. This paper concentrates on the energy management of the HESS side, treating the traction subsystem as an equivalent current source. The traction subsystem interacts with the HESS through the DC link voltage $v_{dc}$ and the traction current $i_{trac}$. The entire vehicle dynamics are represented through the traction current profile applied to the HESS. This paper examines the HESS, focusing on the combination of Li-ion batteries and SCs, connected in active topology, as illustrated in Fig.1. In the active setup, both energy storage units employ bidirectional DC/DC converters. Each converter consists of an inductor attached to the low-voltage end of a power electronics chopper, while its high-voltage end connects to the DC bus.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Battery/SC HESS configuration</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: (a) Battery model (b) SC model</figcaption></figure>

## System modeling and control

* **Battery:** According to [5], the battery can be represented by an equivalent circuit as depicted in Fig. 2(a). In this figure, the transient impedances are ignored, and only the equivalent series resistance, $R_{bat}$, is considered, because the concentration is on the battery's energy characteristics. The battery voltage, $v_{bat}$, consists of the battery's open voltage, $v_{bat_{oc}}$, and the voltage drops across the $R_{bat}$. The $v_{oc}(v_{soc})$ is a non-linear function of $v_{soc}$ which is a representation of the SOC of the battery.

$$v_{bat} = v_{oc}(v_{soc}) - i_{bat}R_{bat} \quad (1)$$

And $v_{bat_{oc}}$ can be found using the following equation.

$$v_{soc} = v_{soc_{init}} - \frac{1}{c_{eq}} \int_{0}^{t} i_{bat} dt \quad (2)$$

$c_{eq}$ shows the total stored charge in the battery, which is the SOC of the battery. This involves translating the battery's nominal capacity from Ahr to charge in coulombs.

* **Supercapacitor:** The modeling of SCs is more straightforward because their voltage directly corresponds to their SOC. As a result, the equivalent circuit model for the SC can be visualized as a large capacitor, $C_{SC}$, in series with an internal resistor $R_{SC}$ as shown in Fig. 2 (b). The mathematical representation of the SC is then defined as:

$$v_{sc} = v_{sc_{init}} - \frac{1}{c_{sc}} \int_{0}^{t} i_{sc} dt - R_{sc} i_{sc} \quad (3)$$

Bi-directional DC/DC converters: In this report, the average model is considered for modeling the DC/DC converters.
<figure><img src="image_4.png" /><figcaption>Fig. 3: Control scheme of the studied HESS</figcaption></figure>

$$
\begin{cases}
v_{ch_x} = m_{ch_x} v_{dc} \\
i_{ch_x} = m_{ch_x} i_x \eta_{ch_x}^{k_{ch_x}} & \text{with } k_{ch_x} = \begin{cases} 1 & \text{if } i_x \ge 0 \\ -1 & \text{if } i_x < 0 \end{cases} \tag{4}
\end{cases}
$$

where "x" stands for either "bat" or "SC",
depending on the sources connected to the
DC/DC converters. The modulation index of the
converter related to the x source is denoted by
$m_{ch_x}$, and $\eta$ represents efficiency.

The DC link voltage which is connected to DC
link capacitor can be obtained from (5).

$$
v_{dc} = \frac{1}{c_{dc}} \int_{0}^{t} (i_{tot} - i_{trac}) dt \quad (5)
$$

where $i_{trac}$ is the traction current applied by the
traction subsystem, and $i_{tot}$ is the total current
formed by the combination of the battery and SCs
subsystems and can be formulated as:

$$
i_{tot} = i_{ch\_bat} + i_{ch\_sc} \quad (6)
$$

• Control strategy: The control strategy of the system is demonstrated in Fig. 3. A Proportional Integral (PI) controller with the compensation of the detected traction current $i_{trac_{mea}}$ is utilized for the DC link voltage control loop. The current of the choppers can be obtained by (7).

$$
i_{x_{ref}} = \frac{i_{ch_{x_{ref}}}}{m_{ch_x} \eta_{ch_x}^{k_{ch_x}}} \quad (7)
$$

The modulation index also can be calculated as:

$$
m_{ch_x} = \frac{v_{ch_{x_{ref}}}}{v_{dc_{mea}}} \qquad (8)
$$

The current distribution block, which is EMS,
employs an RL algorithm to specify the near-
optimal current sharing between the battery and
SC. Table 1 represents the detailed specifications
of the studied HESS composed of a battery bank
and an SC bank.

Reinforcement learning

The RL algorithm is a model-free machine
learning algorithm that does not require any prior
data or the model of the system for training.

<table><caption>Table I: Specification of studied HESS</caption>
  <thead>
    <tr>
      <th>Device</th>
      <th>Specifications</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td rowspan="4">Battery</td>
      <td>Number of modules in series</td>
      <td>3</td>
    </tr>
    <tr>
      <td>Number of banks in parallel</td>
      <td>4</td>
    </tr>
    <tr>
      <td>Module capacity</td>
      <td>110 Ah</td>
    </tr>
    <tr>
      <td>Module nominal voltage</td>
      <td>12.8 V</td>
    </tr>
    <tr>
      <td rowspan="4">Supercapacitor</td>
      <td>Number of modules in series</td>
      <td>3</td>
    </tr>
    <tr>
      <td>Number of banks in parallel</td>
      <td>3</td>
    </tr>
    <tr>
      <td>Module capacitance</td>
      <td>58 F</td>
    </tr>
    <tr>
      <td>Module nominal voltage</td>
      <td>15 V</td>
    </tr>
    <tr>
      <td rowspan="2">DC/DC converter</td>
      <td>Rated power</td>
      <td>30 kW</td>
    </tr>
    <tr>
      <td>Efficiency</td>
      <td>95%</td>
    </tr>
    <tr>
      <td>DC link</td>
      <td>Voltage</td>
      <td>60 V</td>
    </tr>
  </tbody>
</table>

An RL algorithm is composed of two parts: an
agent and an environment. The agent is the
controller and decision-maker and the
environment is what the agent is controlling
which in this case is the HESS.

The RL algorithm is formulated within the Markov Decision Process (MDP) framework, implying that future states depend only on the current state, independent of the system's previous states [22], and the agent and the environment interact through states ($s_t$), actions ($a_t$), and rewards ($r_t$) [23]. In every timestep, the agent receives the states ($s_t \in S$) of the environment, and based on a policy ($\pi$) the agent selects an action among the action space ($a_t \in A$). Then, with the probability of ($P_{s_t,s_{t+1}}^{A_t} = P_r(s_{t+1}|s_t, a_t)$) the environment will transit to the next state ($s_{t+1}$) and the agent will receive a reward ($r_t \in R$) that shows how good the previous action was. After this step, the agent will be updated. This process continues until the agent is learned. The general framework of an RL-based EMS for controlling a HESS in an AEV is illustrated in Fig 4.

This article employs an RL algorithm to perform
the power-sharing between the battery and SC.
The RL algorithms have significant benefits in
controlling the power-sharing in HESS in AEVs.
<figure><img src="image_5.png" /><figcaption>Fig. 4: The general framework of an RL-based EMS of HESS</figcaption></figure>

## Proximal policy optimization (PPO)

PPO is a model-free, on-policy, policy gradient, actor-critic algorithm that is used for RL tasks in both continuous and discrete action spaces. On-policy algorithms use data collected from the current policy to update the policy.

In the actor-critic methods, the actor (policy network) is responsible for selecting the actions, while the critic (value network) evaluates the selected action by the actor by estimating the expected cumulative reward.

The PPO algorithm optimizes the policy using a clipped surrogate objective function, which is designed to stabilize the training process by preventing large, potentially destructive updates to the policy. This characteristic helps PPO algorithm surpasses the other policy gradient methods in terms of stability and efficiency. The main idea in this algorithm is to iteratively update the policy by taking small steps in the direction that the expected reward is improved. The clipped surrogate objective function

The objective function, known as the clipped surrogate objective, is defined as (1).

$$L^{CLIP}(\theta) = E_t[\min(\rho_t(\theta)\hat{A}_t, \text{clip}(\rho_t(\theta), 1 - \varepsilon, 1 + \varepsilon)\hat{A}_t)] \quad (9)$$

where $L^{CLIP}(\theta)$ is the clipped surrogate objective function, $\theta$ represents the parameters of the policy network, $\rho_t(\theta)$ is the probability ratio or importance sampling ratio and $\hat{A}_t$ is the estimated advantage at timestep t, and $\varepsilon$ is a hyperparameter that controls the clipping range. The clipping function constraints $\rho_t(\theta)$ to remain in the range of $[1 - \varepsilon, 1 + \varepsilon]$, ensuring that updates to the policy are small and stable.

The probability ratio formula is shown in (10).

$$\rho_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)} \quad (10)$$

And the advantage function is defined as:

$$\hat{A}_t = \sum_{t'=t}^{T} \gamma^{t'-t} r_{t'} - V(s_t) \quad (11)$$

where $\gamma$ is the discount factor, $r_{t'}$ is the reward at timestep $t'$, and $V(s_t)$ is the value function estimate for state $s_t$. The advantage function determines how good a particular action is in comparison to a baseline (the value function). During the training process, the policy network is updated using the clipped surrogate objective function to optimize the expected cumulative reward and the value network is updated using the mean squared error (MSE) loss, which minimizes the difference between the predicted value $V(s_t; \emptyset)$ and the actual cumulative reward observed from state $s_t$. Fig 5 shows the framework of a PPO algorithm.

The RL-based EMS design is composed of some steps including the identifying the action space, the state space, and reward function. In the following section the MDP model for EMS of a HESS is explained.

In the proposed EMS, the environment is the traction of an AEV, the agent is the PPO agent, the action space is the reference of the battery PI controller ($a = I_{bat}^{\text{ref}}$), which is in the range of $[-50A, 150A]$. The state space is $S = [\text{SoC}_{bat}, \text{SoC}_{SC}, I_{dem}^{\text{tot}}]$, in which $\text{SoC}_{bat}$ and $\text{SoC}_{SC}$ are the state of charge of the battery and the SC, respectively, and $I_{dem}^{\text{tot}}$ is the total current demand of the system. The reward function is also defined so that the EMS can keep the $\text{SoC}_{bat}$ between 20% and 80%, enhance the lifecycle of the battery. For this purpose, the following function is considered as the reward function.

$$R = 2 + r_{bat} + r_{SC} + r_{smooth} \quad (12)$$

The constant reward 2 is used to enhance the mileage. $r_{bat}$ is the reward for keeping the $\text{SoC}_{bat}$ in the desired range as shown in (13).

$$r_{bat} = \begin{cases} 1.5, & \text{if } 20 \le \text{SoC}_{bat} \le 80 \\ -0.02 \times (\text{SoC}_{bat} - 20)^2, & \text{if } \text{SoC}_{bat} < 20 \\ -0.02 \times (\text{SoC}_{bat} - 80)^2, & \text{if } \text{SoC}_{bat} > 80 \end{cases} \quad (13)$$

The $r_{smooth}$ term is also used for increasing the lifecycle of the battery and is defined in (14).

$$r_{smooth} = -0.0015 \times |\Delta \text{SoC}_{bat}| \quad (14)$$

where $\Delta \text{SoC}_{bat}$ is the changes in the $\text{SoC}_{bat}$ compared to the previous timestep.
<figure><img src="image_6.png" /><figcaption>Fig. 5: The general framework of an RL-based EMS of HESS</figcaption></figure>

## Agent training process

In this work, the environment is modeled in MATLAB/Simulink, while the agent is developed and implemented in Python. The environment consists of key components, including the battery, SC, DC/DC converters, load, and controllers. The agent employs the PPO algorithm for training, facilitating effective power management and coordination within the HESS. In order to interface the Python agent with the Simulink model, the Simulink_Gym wrapper, a tool that facilitates the integration of RL algorithms with Simulink models, is used. This wrapper handles the data exchange between a Gym environment in Python and a Simulink model, running in a background instance of Simulink, through TCP/IP communications.

At each timestep, the agent, implemented in Python, sends an action $a_t$ to the Simulink environment. After receiving the $a_t$, the Simulink model computes the next state $s_{t+1}$ and the corresponding reward $r_t$ based on the system's response to the action. This process is repeated at every timestep until the episode ends. Through these episodes, the agent is trained.

## Simulation Results

In order to evaluate the performance of the trained PPO agent, it was implemented in a HESS system that was being used under scaled urban dynamometer driving schedule (UDDS) drive cycle condition depicted in Fig. 6. The performance of the RL-based EMS was then

compared to a rule-based EMS to assess its effectiveness in optimizing the energy utilization, prolonging the battery's lifecycle, and enhancing vehicle mileage. Table II represents the hyperparameters of the trained RL agent.

Fig. 7 and Fig.8 show the SoC variations of the battery and SC storage systems under rule-based and RL-based EMSs, respectively. Based on the results, several key aspects are observed. First, it can be observed that using the RL-based EMS, the battery SoC exhibits a gradual decline in comparison with the rule-based system and maintains a higher battery SoC throughout the drive cycle, which results in extending the battery's lifecycle. On the other hand, the average of the SC SoC is kept higher using the RL-based EMS. The high SoC and frequent usage of SC suggest that the RL-based EMS may be more actively engaging the SC to manage load changes, potentially optimizing energy use more dynamically. Higher average SoC for both battery and SC using the RL-based EMS indicates the potential for increasing the mileage of the AEV.

<figure><img src="image_7.png" /><figcaption>Fig. 6: Power demand of the scaled UDDS drive cycle</figcaption></figure>
<table><caption>**Table II: Hyperparameters of the PPO algorithm**</caption><thead><tr><td>Hyperparameter</td><td>Description</td><td>Value</td></tr></thead><tbody><tr><td>Batch_size</td><td>Size of the minibatch used for each gradient update</td><td>128</td></tr><tr><td>gamma</td><td>Results Discount factor for future rewards</td><td>0.95</td></tr><tr><td>learning_rate</td><td>Optimizer learning rate</td><td>1 × 10<sup>-4</sup></td></tr><tr><td>num_epochs</td><td>Number of epochs for optimizing the surrogate loss per update</td><td>10</td></tr><tr><td>Gae_lmbda</td><td>Generalized advantage estimation trade-off parameter</td><td>0.95</td></tr><tr><td>Total_timestep</td><td>Total number of training timesteps</td><td>500,000</td></tr><tr><td>net_arch</td><td>Network architecture of the policy and value networks</td><td>[128,64]</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Fig. 7: SoC dynamics for the battery and SC under the rule-based EMS</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8: SoC dynamics for the battery and SC under the RL-based EMS</figcaption></figure>

Furthermore, as it is shown in Table III, the battery and the SC hold higher level of SoC at the end of the drive cycle using the RL-based EMS. Although implementing the training code in Python could increase the speed of computation compared to using MATLAB, but still imposes a higher computational burden compared to the

<table><caption>**Table III: Comparison results between the rule-based and RL-based EMSs**</caption><thead><tr><td>EMS Method</td><td>Initial SoC<sub>bat</sub> (%)</td><td>Final SoC<sub>bat</sub> (%)</td><td>Initial SoC<sub>SC</sub> (%)</td><td>Final SoC<sub>SC</sub> (%)</td></tr></thead><tbody><tr><td>Rule-based</td><td>70</td><td>68.69</td><td>80</td><td>50</td></tr><tr><td>RL-based</td><td>70</td><td>69.26</td><td>80</td><td>89</td></tr></tbody></table>

rule-based EMS, which necessitate more powerful hardware.

## Conclusion

In this paper, Simulink_Gym wrapper is used to implement an RL-based EMS the battery and SC energy storage systems. The results have demonstrated the effective application of PPO algorithm for energy management in HESS within AEVs. It is demonstrated that by implementing an RL-based EMS, the power management between batteries and SCs has been significantly improved compared to traditional rule-based EMS. The simulation results confirm that the RL-based approach not only stabilizes the SoC levels across both storage components but also enhances the overall energy efficiency and extends the battery lifespan.

For future studies, the real-world application of these approaches in actual AEVs under diverse conditions can be studied. Additionally, in the future studies, the energy management in the presence of the renewable energy sources can be further explored. These advancements could make AEVs more sustainable and efficient.

## References

[1] S. F. Tie and C. W. Tan, “A review of energy sources and energy management system in electric vehicles,” *Renewable and Sustainable Energy Reviews*, vol. 20, pp. 82–102, Apr. 2013.

[2] “TRAN Appearance: Main Estimates 2023-2024 and Supplementary Estimates (C) 2022-2023,” Government of Canada. Accessed: Sep. 20, 2023. [Online]. Available: https://tc.canada.ca/en/binder/41-zero-emissions-vehicles

[3] P. Ranjbaran, H. Yousefi, G. B. Gharehpetian, and F. R. Astaraei, “A review on floating photovoltaic (FPV) power generation units,” *Renewable and Sustainable Energy Reviews*, vol. 110, pp. 332–347, Aug. 2019.

[4] M. R. Soukhtekouhi, M. Hamzeh, and H. Iman-Eini, “Performance Improvement of Unified Power Quality Conditioner Under Various Load, Source, and Line Conditions Using A New Control Method,” in *2020 11th Power*
*Electronics, Drive Systems, and Technologies Conference (PEDSTC)*, IEEE, Feb. 2020, pp. 1-
6.

[5] B.-H. Nguyen, T. Vo-Duy, M. C. Ta, and J. P. F. Trovao, "Optimal Energy Management of Hybrid Storage Systems Using an Alternative Approach of Pontryagin's Minimum Principle," *IEEE Transactions on Transportation Electrification*, vol. 7, no. 4, pp. 2224-2237, Dec. 2021.

[6] H. Li, A. Ravey, A. N. Diaye, and A. Djerdir, "Online adaptive equivalent consumption minimization strategy for fuel cell hybrid electric vehicle considering power sources degradation," *Energy Convers Manag.*, vol. 192, no. March, pp. 133-149, 2019.

[7] R. Xiong, H. Chen, C. Wang, and F. Sun, "Towards a smarter hybrid energy storage system based on battery and ultracapacitor - A critical review on topology and energy management," *J Clean Prod.*, vol. 202, pp. 1228-1240, Nov. 2018.

[8] P. Ranjbaran, J. Ebrahimi, A. Bakhshai, and P. Jain, "Reinforcement Learning-Based Approaches to Energy Management of Hybrid Energy Storage Systems in Electric Vehicles," in *2023 IEEE 14th International Conference on Power Electronics and Drive Systems (PEDS)*, IEEE, Aug. 2023, pp. 1-6.

[9] S. Shahnooshi, P. Ranjbaran, J. Ebrahimi, A. Bakhshai, and P. Jain, "Reinforcement Learning-based Control of a Buck Converter: A Comparative Study of DQN and DDPG Algorithms," in *2023 25th European Conference on Power Electronics and Applications (EPE'23 ECCE Europe)*, IEEE, Sep. 2023, pp. 1-10.

[10] P. Ranjbaran, S. Shahnooshi, J. Ebrahimi, A. Bakhshai, and P. Jain, "Exploring the Effectiveness of Different State Spaces and Reward Functions in Reinforcement Learning-based Control of a DC/DC Buck Converter," in *2023 25th European Conference on Power Electronics and Applications (EPE'23 ECCE Europe)*, IEEE, Sep. 2023, pp. 1-10.

[11] D. Xu, Y. Cui, J. Ye, S. W. Cha, A. Li, and C. Zheng, "A soft actor-critic-based energy management strategy for electric vehicles with hybrid energy storage systems," *J Power Sources*, vol. 524, p. 231099, Mar. 2022.

[12] N. Yang, L. Han, R. Liu, Z. Wei, H. Liu, and C. Xiang, "Multiobjective Intelligent Energy Management for Hybrid Electric Vehicles Based on Multiagent Reinforcement Learning," *IEEE Transactions on Transportation Electrification*, vol. 9, no. 3, pp. 4294-4305, Sep. 2023.

[13] Y. Xu, G. Han, D. Zhang, and J. Wei, "Joint Energy Management and Eco-Routing for Electric Vehicles with Hybrid Energy Storage

Systems," in 2023 4th Information
Communication Technologies Conference
(ICTC), IEEE, May 2023, pp. 374-378.

[14] F. Li et al., "Incentive learning-based energy management for hybrid energy storage system in electric vehicles," *Energy Convers Manag.*, vol. 293, p. 117480, Oct. 2023.

[15] Y. Wu et al., "Driving style-aware energy management for battery/supercapacitor electric vehicles using deep reinforcement learning," *J Energy Storage*, vol. 73, p. 109199, Dec. 2023.

[16] Y. Ye, H. Wang, B. Xu, and J. Zhang, "An imitation learning-based energy management strategy for electric vehicles considering battery aging," *Energy*, vol. 283, Nov. 2023.

[17] Z. Fu, H. Wang, F. Tao, B. Ji, Y. Dong, and S. Song, "Energy Management Strategy for Fuel Cell/Battery/Ultracapacitor Hybrid Electric Vehicles Using Deep Reinforcement Learning With Action Trimming," *IEEE Trans Veh Technol.*, vol. 71, no. 7, pp. 7171-7185, Jul. 2022.

[18] H. Hu, C. Lu, J. Tan, S. Liu, and D. Xuan, "Effective energy management strategy based on deep reinforcement learning for fuel cell hybrid vehicle considering multiple performance of integrated energy system," *Int J Energy Res.*, vol. 46, no. 15, pp. 24254-24272, Dec. 2022.

[19] J. Wang, J. Zhou, and W. Zhao, "Deep reinforcement learning based energy management strategy for fuel cell/battery/supercapacitor powered electric vehicle," *Green Energy and Intelligent Transportation*, vol. 1, no. 2, p. 100028, Sep. 2022.

[20] Y. Li, J. Tao, and K. Han, "Rule and Q-learning based Hybrid Energy Management for Electric Vehicle," in *2019 Chinese Automation Congress (CAC)*, IEEE, Nov. 2019, pp. 51-56.

[21] R. Xiong, J. Cao, and Q. Yu, "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," *Appl Energy*, vol. 211, no. 5, pp. 538-548, Feb. 2018.

[22] A. Hannegudda and B. Xu, "A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution," *Renewable and Sustainable Energy Reviews*, vol. 154, no. October 2021, p. 111833, 2022.

[23] D. Xu, Y. Cui, J. Ye, S. Won, A. Li, and C. Zheng, "A soft actor-critic-based energy management strategy for electric vehicles with hybrid energy storage systems," *J Power Sources*, vol. 524, no. December 2021, p. 231099, 2022.