# A Multi-Agent Reinforcement Learning-based Secondary Control for Voltage Restoration and Current Sharing in DC Microgrids Keywords

Shima Shahnooshi, Javad Ebrahimi, Alireza Bakhshai

► To cite this version:

Shima Shahnooshi, Javad Ebrahimi, Alireza Bakhshai. A Multi-Agent Reinforcement Learning-based Secondary Control for Voltage Restoration and Current Sharing in DC Microgrids Keywords. The 26th European Conference on Power Electronics and Applications, GDR SEEDS France & EPE Association, Mar 2025, Paris, France. 10.34746/epe2025-0120. hal-05095998

HAL Id: hal-05095998

https://utc.hal.science/hal-05095998v1

Submitted on 3 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Multi-Agent Reinforcement Learning-based Secondary Control for Voltage Restoration and Current Sharing in DC Microgrids

Shima Shahnooshi

Department of Electrical and
Computer Engineering
Queen's University
Kingston, Canada

Shima.shahnooshi@queensu.ca

Javad Ebrahimi

Department of Electrical
and Computer Engineering
Queen's University
Kingston, Canada

javad.ebrahimi@queensu.ca

Alireza Bakhshai

Department of Electrical and
Computer Engineering
Queen's University
Kingston, Canada

alireza.bakhshai@queensu.ca

## Keywords

AC-DC microgrid, Droop control, Reinforcement
Learning, Multi-agent system, Secondary
Control, Voltage control.

## Abstract

Microgrids are designed for sustainable energy generation and the efficient use of Distributed Generators (DGs). DC microgrids face challenges such as current sharing and voltage restoration, commonly addressed using droop control. Despite its simplicity, droop control can lead to voltage deviations and inefficient current sharing. This paper develops a Multi-Agent Reinforcement Learning (MARL) based secondary control method using Deep Q-Networks (DQN) to improve voltage restoration and current sharing in DC microgrids. The proposed method leverages centralized training to utilize comprehensive environmental information, integrating both the voltage restoration and current sharing objectives into each agent's local reward function. Simulation results demonstrate the effectiveness of the MARL approach in addressing the limitations of droop control, ensuring stable current flow and enhancing overall system performance.

## Introduction

Microgrids are designed for sustainable energy generation and the efficient use of Distributed Generators (DGs) [1]. These resources can be integrated into a common AC or DC grid via power electronic interfaces, offering flexibility in conversion and power levels. DC microgrids improve system efficiency and reliability and simplify the integration of renewable energy

systems (RES) and energy storage systems (ESS) due to their inherent DC output [2], [3]. Furthermore, they do not require synchronization or reactive power. However, DC microgrids face challenges like those of AC microgrids, such as current sharing and voltage restoration. Addressing these challenges requires reliable control systems to ensure stable power flow and overall system stability [4].

A common approach to perform current sharing in DC microgrids is droop control. This technique allows for decentralized control by adjusting the output voltage of power sources in proportion to their load current [5]. Droop control has the advantage of being simple and easy to implement, as it does not require communication between different DGs, enhancing the robustness and flexibility of the system. However, there are certain drawbacks to consider. Droop control can lead to voltage deviations, which may adversely affect the performance of sensitive equipment. Moreover, without proper tuning, it can result in inefficient power sharing and potential instability in the microgrid. To address these issues, it is often necessary to integrate supplementary control strategies to fine-tune the performance of the microgrid.

Building on the inherent challenges of voltage deviation and current sharing in droop control, numerous studies have explored various methods to improve system performance and reliability. Authors in [6] introduced a droop index to control the circulating current and current sharing difference. Considering the challenges associated with conventional droop, [7] proposes a feedback linearized method that does not require a detailed model of the system. An approach to restore voltage and ensure proper current sharing is adding a compensation term to the outer stage
of the primary control. Authors in [7] add a voltage

compensation term as a function of DG terminal voltage to compensate for the voltage drop on the line resistance. However, it does not consider the voltage drop due to droop control. In [8], a low-bandwidth communication-based droop control is proposed. A PI controller is used to decrease the difference between the average voltage of DG terminals and the reference value. The compensation term is added to the outer stage of the primary control. The proposed method in [9] combines a communication-free control as the secondary control and a distributed control as the tertiary control. The authors use neural networks to adaptively change the droop coefficient and voltage compensation term, which are multiplied by the feedback of load current. However, supervised learning requires large training datasets to consider all the possible conditions of the system [10].

Until now, previous works have required detailed system models to implement accurate current sharing and DC bus voltage restoration [11]. Additionally, linear control techniques are affected by the operating conditions. Since the components of DC microgrids, such as RESs, are inherently nonlinear, it is impractical to obtain an accurate system model in many cases, and therefore, it is not suitable to use linear controllers [12].

Reinforcement Learning (RL) has recently attracted considerable attention from researchers for its applications in power systems and power electronics [13], [14], [15]. One of RL's key advantages is its ability to operate without requiring detailed system models. Simultaneously, MARL has proven to be valuable in various fields by managing complex, distributed tasks through decentralized and collaborative efforts. In MARL, multiple agents function autonomously while interacting with each other to achieve common goals. This approach provides several benefits, such as improved scalability, flexibility, and robustness.

MARL has been explored in previous studies for distributed frequency control of islanded microgrids [16], controlling a DC solid-state transformer [17], energy management of distribution systems [18], and the identification and remediation of FDI attacks [19]. In [20], a MARL with centralized training and decentralized execution was proposed for Volt/Var control in inverter-based systems. MARL has also been

applied to ensure voltage restoration and current sharing. Authors in [21] utilize a distributed consensus method and a predefined pinning consensus value to maximize the global reward of a MARL.

Inspired by the aforementioned studies, this work develops a MARL-based secondary control to coordinate voltage restoration and current sharing in DC microgrids. The focus of the proposed method is on voltage compensation caused by droop control at the primary control level. To ensure that current sharing accuracy is not compromised, the current sharing objective is incorporated into the local reward function of each agent. The agents are implemented using the DQN algorithm and trained in a centralized manner. This centralized training allows the RL agents to utilize comprehensive environmental information, including observations and actions. Furthermore, by obtaining the average of terminal voltages, the voltage error between the reference value and average voltage is included in the state space and considered in the reward function, adding another level of coordination between the parallel DGs in the DC microgrid.

The remainder of the paper is organized as follows. Section II describes the DC microgrid system and the principles of droop control. Section III reviews the basics of the DQN algorithm and MARL, and it develops the proposed MARL secondary control method to achieve voltage restoration and current sharing in DC microgrids. Section IV presents simulation results that demonstrate the effectiveness of the proposed MARL method in restoring voltage and maintaining current sharing in a simulated DC microgrid. Finally, the conclusion is presented in Section V.

## System Description

Figure 1 shows a simple structure of a DC microgrid with two DGs feeding loads. The power on the source side can be generated by various types of RES and ESS. Each DG is connected to the main DC bus through a DC-DC converter, which processes the generated power to meet the load requirements. In DC microgrids with parallel DGs, the main control objectives are current sharing and DC bus voltage control. These objectives are achieved through primary control level, which includes current control loop, voltage control loop, and droop control. This level of control ensures the proper duty ratios of the converters in a decentralized manner.
<figure><img src="image_2.png" /><figcaption>Figure 1. A two-DG DC microgrid with primary control.</figcaption></figure>

## Droop Control in DC Microgrids

Droop control is a widely adopted method for ensuring equitable current and power sharing among parallel power units in DC microgrids, which is described as

$$V_{ref,i} = V_{dc}^* - R_{d,i} I_{dc,i}, i \in N \quad (1)$$

where i represents the number of the DG, N is the total number of DGs, $V_{ref,i}$ is the voltage reference of the i^th^ DG terminal, $V_{dc}^*$ is the nominal DC voltage, $I_{dc,i}$ is the output current of the corresponding DG, and $R_{di}$ is the droop gain of the converter. The droop control operates by dynamically reducing the reference voltage $V_{ref,i}$ as the output current $I_{dc,i}$ increases, thereby facilitating effective load sharing among the parallel DGs.

To ensure that the output voltage falls within the allowable range, $R_{d,i}$ is calculated based on the maximum voltage deviation $\Delta V_i^{max}$ and maximum output current $I_{dc,i}^{max}$ of the i^th^ DG as follows:

$$R_{d,i} = \frac{\Delta V_i^{max}}{I_{dc,i}^{max}}, i \in N \quad (2)$$

where $\Delta V_i^{max}$ is set to the 5% of the nominal voltage $V_{dc}^*$. In addition to the voltage drop caused by the inherent characteristics of droop control, the total voltage drop at bus terminal voltage $V_{bus}$ is also influenced by line resistance. The bus voltage $V_{bus}$ is calculated as follows:

$$V_{bus} = V_{ref,i} - R_{t,i} I_{dc,i} \\ = V_{dc}^* - (R_{d,i} + R_{t,i}) I_{dc,i} \quad (3)$$

where $R_{t,i}$ is the line resistance of i^th^ DG.

According to (3) and the DC microgrid shown in Figure 1, the DC output current of each converter is inversely proportional to the droop gain which is described as follows:

$$\frac{I_{dc,1}}{I_{dc,2}} = \frac{R_{d,2} + R_{t,2}}{R_{d,1} + R_{t,1}} \quad (4)$$

In small systems, the line resistance is often negligible because the droop gain is typically set much higher than the line resistance to achieve more accurate current sharing. However, in larger systems, the line resistance significantly affects the accuracy of current sharing. Therefore, it must be taken into account and compensated for by auxiliary control techniques.

## DC Voltage Restoration

In addition to current sharing, voltage restoration is another challenge of droop control, which is the main focus of this work. As reviewed in the Introduction, one of the ways to compensate the voltage drop and ensure voltage regulation is adding a voltage compensation term to the outer level of the primary control. As a result, the terminal reference voltage is adjusted as

$$V_{ref,i} = V_{dc}^* - R_{d,i} I_{dc,i} + \Delta V_i \quad (5)$$

where $\Delta V_i$ is the compensation term designed and fedforward to the outer level of the primary control level using a secondary control method. In this study, the correction term $\Delta V_i$ is adaptively designed based on a MARL-based control method and is fedforward to the decentralized controller of each converter. The proposed secondary method will be described in the following section.

## MARL Voltage Restoration and Current Sharing Method

In MARL, multiple agents interact with the environment, each employing RL techniques to iteratively learn their policies. MARL can be approached through either decentralized or centralized frameworks. In decentralized learning, each agent is trained independently, simplifying system design but resulting in a non-stationary environment. Conversely, in centralized learning, a high-level process collects the experiences of all agents to learn a comprehensive policy, thereby creating a stationary environment.

MARL can be implemented using various RL algorithms. This paper explains the principles
of DQN and develops a secondary voltage restoration method using a multi-agent DQN approach.

## Algorithm I MARL based on DQN Method

**Initialize:**

Replay memory *D* with capacity *N*

Q-network $Q_i$ with random weights $\theta_i$ for each agent *i*

Target Q-network $Q_i^-$ with weights $\theta_i^-$ = $\theta_i$

**Training Loop:**

**for** each episode from 1 to *M* **do**

Initialize state *s*

**for** each time step *t* from 1 to *T* **do**

**for** each agent *i* in parallel **do**

Observe local state $s_{i,t}$

Select action $a_{i,t}$ using an epsilon-greedy policy:

With probability $\epsilon$ select a random action

Otherwise $a_{i,t} = \underset{a}{\operatorname{argmax}} Q(s_{i,t}, a; \theta_i)$

Execute action $a_{i,t}$ in the environment

Observe reward $r_{i,t}$ and next state $s_{i,t+1}$

Store transition $(s_{i,t}, a_{i,t}, r_{i,t}, s_{i,t+1})$ in replay memory *D*

**end for**

**end for**

**for** each agent *i* in parallel **do**

Sample a random minibatch of transitions from *D*

Set $y_{i,j} =$

$\begin{cases} r_{i,j}, & \text{Terminal } s_{i,j+1} \\ r_{i,j} + \gamma \max_{a'} Q^-(s_{i,j+1}, a'; \theta_i^-), & \text{Non-terminal } s_{i,j+1} \end{cases}$

Perform a gradient descent step on $(y_{i,j} - Q_i(s_j, a_j; \theta_i))^2$

Periodically update $\theta_i^- = \eta\theta_i^- + (1 - \eta)\theta_i$

**end for**

**end for**

**Output:**

Trained Q-networks $Q_i$ for each agent *i*

## Principles of DQN Algorithm

DQN combines Q-learning with deep neural networks to manage high-dimensional state spaces. Q-learning is a model-free RL algorithm designed to find the optimal action-value function $Q^*(s, a)$ for a given Markov Decision Process (MDP). The action-value function $Q(s, a)$ represents the expected cumulative reward for taking action *a* in state *s* and following the optimal policy thereafter.

To address the limitations of Q-learning in high-dimensional state spaces, DQN approximates the Q-function using a deep neural network. The neural network $Q(s, a, \theta)$ takes the state *s* as input and outputs Q-values for all possible actions *a*, where *θ* represents the network parameters [22].

<figure><img src="image_3.png" /><figcaption>Figure 2. General framework of the proposed MARL secondary control.</figcaption></figure>

The DQN training process involves minimizing the loss function:

$$L(\theta) = \mathbb{E}_{(s,a,r,s')[(y - Q(s, a; \theta))^2]} \quad (6)$$

where *r* is the reward received after taking action *a*, *s'* is the next state, and *y* is the target Q-value, defined as:

$$y = r + \gamma \max_{a'} Q(s', a'; \theta^-) \quad (7)$$

Here, $\theta^-$ are the parameters of the target network, which are periodically updated to match $\theta$.

Algorithm I elaborates the details of the centralized MARL based on DQN algorithm. The pseudocode outlines a Multi-Agent DQN with Centralized Training. It initializes the replay memory, Q-networks, and target Q-networks. During training, each agent observes its state, selects actions via an epsilon-greedy policy, and stores the resulting transitions in a shared replay memory. Agents then sample from this memory to update their Q-networks, with target networks periodically updated for stability. This centralized approach enhances coordination by using collective experiences, addressing non-stationarity, and producing trained Q-networks for each agent.
## Proposed MARL Secondary Control

Figure 2 shows the multi-agent voltage restoration method for each DG. Each converter uses decentralized control at the primary control level, where droop control causes voltage deviation at the output terminal, requiring compensation using secondary control. The voltage correction term, according to (5), is added to the outer level of primary control. This paper designs a MARL voltage restoration method based on the DQN algorithm to provide the correction term $\Delta V_i$ for each converter's primary control. The framework of this MARL method will be elaborated in the following.

### A. Action Space

The action of the $i^{th}$ agent is the correction term $\Delta V_i$ of the $i^{th}$ converter, as described in (5). The range of this action should be designed based on the maximum allowable voltage deviation, according to (2). $I_{dc,i}^{max}$ is obtained according to the maximum power of each DG, $P_{dc,i}^{max}$. The DC microgrid in this study has a maximum load power of 10 kW. Assuming that the two DGs in Figure 1 each supply half of the maximum power, $P_{dc,i,max}$ is 5 kW for each converter, and $I_{dc,i}^{max}$ is defined as

$$I_{dc,i}^{max} = \frac{P_{dc,i}^{max}}{V_{dc}^{*}} \qquad (8)$$

which is equal to 22A in this study. $R_{d,i}$ is also designed to be 0.5 for both converters, and consequently, $\Delta V_i^{max}$ is set to 11 for the designed system, which is below %5 of the nominal DC voltage.

The action range is set between 0 and 15, with each action separated by 0.5 units. The action space can be expressed as $A_i = \{0, 0.5, 1, ..., 14.5, 15\}$.

### B. State Space

For a DC microgrid with a number of N DGs, the states of the i-th RL agent are the terminal voltage $V_{dc,i}$ and the output current $I_{dc,i}$ of the corresponding DG, the global voltage error $e_V$ and the global current error $e_I$. The global voltage error $e_V$ is the difference between the reference DC bus voltage and the average of DC terminal voltages of DGs, which is defined as

$$e_V = \left| V_{dc}^* - \frac{\sum_{n=1}^{N} V_{dc,n}}{N} \right| \qquad (9)$$

where N is the total number of DGs in the DC microgrid. Assuming the DGs have similar power ratings, they will supply equal currents to the load. Consequently, the droop coefficients in the primary controllers of the converters are set to be identical, and $e_I$ is defined as

$$e_I = \sum_{n=2}^{N} \left| R_{d1} I_{dc,1} - R_{dn} I_{dc,n} \right| \qquad (10)$$

For the DC microgrid shown in Figure 1, which includes two DGs, $e_V$ is defined as $\left| V_{dc}^* - \frac{V_{dc,1} + V_{dc,2}}{2} \right|$ and $e_I$ is defined as $\left| R_{d,1} I_{dc,1} - R_{d,2} I_{dc,2} \right|$.

### C. Reward Function

A multi-objective reward function is defined in this study to accomplish DC bus voltage restoration and current sharing accuracy. It should be noted that the focus of this study is voltage restoration by adding a compensation term to the primary control level, and the current sharing accuracy can be fulfilled through different approaches, such as adaptive droop and nonlinear droop characteristics [3], [23]. However, the current sharing accuracy term in the reward function ensures that the voltage restoration does not compromise the accuracy of current sharing among DGs.

The instant voltage and current rewards have a negative relationship to the voltage and current errors, respectively, and are defined as

$$\begin{aligned} r_V &= 10 - |e_V|(\text{if } (|e_V| \le 2)) + 1 \\ &\quad - 10|e_V|(\text{if } (|e_V| > 2)) \\ r_I &= 5 - |e_I|(\text{if } (|e_I| \le 0.1)) + 2 \\ &\quad - 5|e_I|(\text{if } (|e_I| > 0.1)) \end{aligned} \qquad (11)$$

where $r_V$ is the voltage reward and $r_I$ is the current reward. The total reward is the sum of $r_V$ and $r_I$ and is defined as $r_T = r_V + r_I$.

### D. Critic Network and Hyperparameters

The Critic network architecture for the DQN agents features an input layer corresponding to the state count and an output layer with 30 neurons, representing the discrete actions. This architecture uses the Rectified Linear Unit (ReLU) activation function. Adding two hidden layers, each with 128 neurons, significantly improves performance by enhancing steady-state and dynamic responses.
Consequently, the Critic network is designed with three 256-neuron hidden layers. Details on the hyperparameters for the DQN agents can be found in Table I.

<table><caption>Table I: Hyperparameters of the DQN agents</caption><thead><tr><th>Hyperparameter</th><th>Value</th></tr></thead><tbody><tr><td>Discount factor (γ)</td><td>0.9</td></tr><tr><td>Learning rate (α)</td><td>0.001</td></tr><tr><td>Epsilon greedy exploration<br>(Epsilon, EpsilonMin, EpsilonDecay)</td><td>(0.9, 0.01, 0.005)</td></tr><tr><td>Target update frequency</td><td>4</td></tr><tr><td>Experience buffer length</td><td>100000</td></tr><tr><td>Minibatch size</td><td>512</td></tr></tbody></table>

## Simulation Results

The islanded DC microgrid with two parallel DGs, as shown in Figure 1, is simulated in MATLAB Simulink. The proposed MARL agents, as shown in Figure 2, are trained in MATLAB to demonstrate the effectiveness of the proposed secondary controller. The parameters of the simulated DC microgrid are listed in Table II.

<table><caption>TABLE II. Electrical and Control Parameters</caption><tbody><tr><th colspan="3">Electrical Parameters for i=1,2</th></tr><tr><th>Parameter</th><th>Symbol</th><th>Value</th></tr><tr><td>Input voltage</td><td>V<sub>in</sub></td><td>400 V</td></tr><tr><td>Nominal DC-link voltage</td><td>V<sub>bus</sub></td><td>230 V</td></tr><tr><td>Buck converter inductor</td><td>L<sub>i</sub></td><td>1.5 mH</td></tr><tr><td>Buck converter capacitor</td><td>C<sub>i</sub></td><td>940 uF</td></tr><tr><td>Filter inductor</td><td>L<sub>ti</sub></td><td>1 mH</td></tr><tr><td>Filter capacitor</td><td>C<sub>ti</sub></td><td>1.2mF</td></tr><tr><td>Line resistance</td><td>R<sub>ti</sub></td><td>0.1 Ω</td></tr><tr><td>Switching Frequency</td><td>f<sub>swi</sub></td><td>20 kHz</td></tr><tr><th colspan="3">Control Parameters for i=1,2</th></tr><tr><td>Proportional and integral gains of voltage control</td><td>k<sub>Pv</sub>, k<sub>Iv</sub></td><td>2, 100</td></tr><tr><td>Proportional and integral gains of current control</td><td>k<sub>PI</sub>, k<sub>Ii</sub></td><td>5, 20</td></tr><tr><td>Droop gain</td><td>R<sub>di</sub></td><td>0.5</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Figure 3. DC-link voltage of the proposed MARL secondary control for a step change in resistive load at t=0.5s and a step-change in CPL demand at t=1s.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4. DG currents of the proposed MARL secondary control for a step change in resistive load at t=0.5s and a step-change in CPL demand at t=1s.</figcaption></figure>

Figure 3 shows the DC-link voltage for two step changes in load power. At t = 0.5s, the amplitude of the resistive load changes from 25 Ω to 50 Ω, and the load demand changes at t = 1s from 1 kW to 3 kW. It can be seen that the DC-link voltage remains regulated around the nominal value as the MARL secondary control is able to compensate for the voltage drop.

Figure 4 shows DG currents for two step changes in load power at t = 0.5s and t = 1s. It can be seen that the proposed MARL secondary control ensures proper current sharing among DGs.

To evaluate the effectiveness of the proposed secondary control method in comparison to conventional approaches, the classic secondary control method described in [8] was also simulated in MATLAB Simulink. For a fair comparison, the sampling time of the PI controller at the secondary control level was discretized to 0.03 seconds, matching the sampling time used for the RL-based control method. Figures 5 and 6 show the DC-link voltages and DG currents of the simulated DC microgrid using the classic method. While the conventional method achieves proper current sharing and DC voltage regulation, the RL-based secondary control demonstrates a faster dynamic response, both during activation and under step changes in load.
<figure><img src="image_6.png" /><figcaption>Figure 5. DC-link voltage of the classic secondary control for a step change in resistive load at t=0.5s and a step-change in CPL demand at t=1s.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6. DG currents of the classic secondary control for a step change in resistive load at t=0.5s and a step-change in CPL demand at t=1s.</figcaption></figure>

## Conclusion

This paper developed a MARL-based secondary control method using DQN to address voltage restoration and current sharing challenges in DC microgrids. Centralized training was employed, integrating voltage restoration and current sharing objectives into each agent's local reward function. Simulation results validated the approach, showing significant improvements over traditional droop control methods. The MARL-based strategy effectively ensures stable and efficient DC microgrid operation, contributing to the advancement of reinforcement learning applications in power systems. Future research may explore its application to more complex microgrid configurations.

## References

[1] T. Dragicevic, X. Lu, J. C. Vasquez, and J. M. Guerrero, "DC Microgrids - Part I: A Review of Control Strategies and Stabilization Techniques," Jul. 01, 2016, *Institute of Electrical and Electronics Engineers Inc.* doi: 10.1109/TPEL.2015.2478859.

[2] X. K. Liu, S. Q. Wang, M. Chi, Z. W. Liu, and Y. W. Wang, "Resilient Secondary Control and Stability Analysis for DC Microgrids under Mixed Cyber Attacks," *IEEE Transactions on Industrial Electronics*, vol. 71, no. 2, pp. 1938-1947, Feb. 2024, doi: 10.1109/TIE.2023.3262893.

[3] P. Prabhakaran, Y. Goyal, and V. Agarwal, "Novel nonlinear droop control techniques to overcome the load sharing and voltage regulation issues in DC Microgrid," *IEEE Trans Power Electron*, vol. 33, no. 5, pp. 4477-4487, May 2018, doi: 10.1109/TPEL.2017.2723045.

[4] P. Wang, M. Zaery, D. Zhao, W. Wang, and D. Xu, "Combined Control Strategy for Proportional Current Sharing in DC Microgrid Clusters," *IEEE Transactions on Industrial Electronics*, vol. 70, no. 11, pp. 11251-11261, Nov. 2023, doi: 10.1109/TIE.2022.3225852.

[5] A. Amanipoor, M. S. Golsorkhi, N. Bayati, and M. Savaghebi, "V-Iq Based Control Scheme for Mitigation of Transient Overvoltage in Distribution Feeders with High PV Penetration," *IEEE Trans Sustain Energy*, vol. 14, no. 1, pp. 283-296, Jan. 2023, doi: 10.1109/TSTE.2022.3211179.

[6] S. Augustine, M. K. Mishra, and N. Lakshminarasamma, "Adaptive droop control strategy for load sharing and circulating current minimization in low-voltage standalone DC microgrid," *IEEE Trans Sustain Energy*, vol. 6, no. 1, pp. 132-141, Jan. 2015, doi: 10.1109/TSTE.2014.2360628.

[7] A. Tah and D. Das, "An Enhanced Droop Control Method for Accurate Load Sharing and Voltage Improvement of Isolated and Interconnected DC Microgrids," *IEEE Trans Sustain Energy*, vol. 7, no. 3, pp. 1194-1204, Jul. 2016, doi: 10.1109/TSTE.2016.2535264.

[8] X. Lu, J. M. Guerrero, K. Sun, and J. C. Vasquez, "An improved droop control method for dc microgrids based on low bandwidth communication with dc bus voltage restoration and enhanced current sharing accuracy," *IEEE Trans Power Electron*, vol. 29, no. 4, pp. 1800-1812, 2014, doi: 10.1109/TPEL.2013.2266419.

[9] P. Wang, M. Zaery, D. Zhao, W. Wang, and D. Xu, "Combined Control Strategy for Proportional Current Sharing in DC Microgrid Clusters," *IEEE Transactions on Industrial Electronics*, vol. 70, no. 11, pp. 11251-11261, Nov. 2023, doi: 10.1109/TIE.2022.3225852.

[10] H. Hussaini, T. Yang, G. Bai, M. Urrutia-Ortiz, and S. Bozhko, "Artificial Intelligence-Based Hierarchical Control Design for Current Sharing and Voltage Restoration in DC Microgrid of the More Electric Aircraft," *IEEE Transactions on Transportation Electrification*, vol. 10, no. 1, pp. 566-582, Mar. 2024, doi: 10.1109/TTE.2023.3289773.

[11] Y. Yu, G. P. Liu, Y. Huang, and J. M. Guerrero, "Distributed Data-Driven Secondary Regulation for the Conflict Between Voltage Recovery and Accurate Current Sharing in DC Microgrids," *IEEE*
*Trans Power Electron*, vol. 38, no. 8, pp. 9617-9634, Aug. 2023, doi: 10.1109/TPEL.2023.3280164.

[12] S. Shahnooshi, J. Ebrahimi, and A. Bakhshai, "A Comparative Study of Reinforcement Learning and Classic Nonlinear Methods for Stabilization of DC Microgrids Supplying Constant Power Loads," in *Proceedings of the International Conference on Power Electronics and Drive Systems*, Institute of Electrical and Electronics Engineers Inc., 2023. doi: 10.1109/PEDS57185.2023.10246497.

[13] Institute of Electrical and Electronics Engineers, 2023 25th European Conference on Power Electronics and Applications (EPE'23 ECCE Europe) 4-8 Sept. 2023: conference location: Aalborg, Denmark.

[14] Institute of Electrical and Electronics Engineers, 2023 25th European Conference on Power Electronics and Applications (EPE'23 ECCE Europe) 4-8 Sept. 2023: conference location: Aalborg, Denmark.

[15] P. Ranjbaran, J. Ebrahimi, A. Bakhshai, and P. Jain, "Reinforcement Learning-Based Approaches to Energy Management of Hybrid Energy Storage Systems in Electric Vehicles," in *Proceedings of the International Conference on Power Electronics and Drive Systems*, Institute of Electrical and Electronics Engineers Inc., 2023. doi: 10.1109/PEDS57185.2023.10246568.

[16] M. Adibi and J. Van Der Woude, "Secondary Frequency Control of Microgrids: An Online Reinforcement Learning Approach," *IEEE Trans Automat Contr*, vol. 67, no. 9, pp. 4824-4831, Sep. 2022, doi: 10.1109/TAC.2022.3162550.

[17] Y. Zeng et al., "Deep Reinforcement Learning-Enabled Distributed Uniform Control for a DC Solid State Transformer in DC Microgrid," *IEEE Transactions on Industrial Electronics*, vol. 71, no. 6, pp. 5818-5829, Jun. 2024, doi: 10.1109/TIE.2023.3294584.

[18] H. S. V. S. K. Nunna, A. Sesetti, A. K. Rathore, and S. Doolla, "Multiagent-Based Energy Trading Platform for Energy Storage Systems in Distribution Systems with Interconnected Microgrids," *IEEE Trans Ind Appl*, vol. 56, no. 3, pp. 3207-3217, May 2020, doi: 10.1109/TIA.2020.2979782.

[19] A. J. Abianeh, Y. Wan, F. Ferdowsi, N. Mijatovic, and T. Dragicevic, "Vulnerability Identification and Remediation of FDI Attacks in Islanded DC Microgrids Using Multiagent Reinforcement Learning," *IEEE Trans Power Electron*, vol. 37, no. 6, pp. 6359-6370, Jun. 2022, doi: 10.1109/TPEL.2021.3132028.

[20] A. Amanipoor and M. S. Golsorkhi, "Stability Analysis and Design of Volt-VAR Controller for Grid Connected PV Systems with Consideration of the Impact of Voltage Feedforward," in *IECON Proceedings (Industrial Electronics Conference)*, IEEE Computer Society, 2022. doi: 10.1109/IECON49645.2022.9968456.

[21] Z. Liu, Y. Luo, R. Zhuo, and X. Jin, "Distributed reinforcement learning to coordinate current sharing and voltage restoration for islanded DC microgrid," *Journal of Modern Power Systems and Clean Energy*, vol. 6, no. 2, pp. 364-374, Mar. 2018, doi: 10.1007/s40565-017-0323-y.

[22] S. Shahnooshi, J. Ebrahimi, and A. Bakhshai, "A Reinforcement Learning Controller Based on Double DQN for DC microgrids with Constant Power Loads," in *2024 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)*, IEEE, Aug. 2024, pp. 376-380. doi: 10.1109/CCECE59415.2024.10667207.

[23] S. Augustine, M. K. Mishra, and N. Lakshminarasamma, "Adaptive droop control strategy for load sharing and circulating current minimization in low-voltage standalone DC microgrid," *IEEE Trans Sustain Energy*, vol. 6, no. 1, pp. 132-141, Jan. 2015, doi: 10.1109/TSTE.2014.2360628.