# Intelligent Integration of Renewable Energy Resources Review: Generation and Grid Level Opportunities and Challenges

Aras Ghafoor, Jamal Aldahmashi, Judith Apsley, Siniša Djurović, Xiandong Ma, Mohamed Benbouzid

► To cite this version:

Aras Ghafoor, Jamal Aldahmashi, Judith Apsley, Siniša Djurović, Xiandong Ma, et al. Intelligent Integration of Renewable Energy Resources Review: Generation and Grid Level Opportunities and Challenges. Energies, 2024, 17 (17), pp.4399. 10.3390/en17174399. hal-05340859

HAL Id: hal-05340859

https://hal.univ-brest.fr/hal-05340859v1

Submitted on 31 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Review

# Intelligent Integration of Renewable Energy Resources Review: Generation and Grid Level Opportunities and Challenges

Aras Ghafoor <fnref n="1" />, Jamal Aldahmashi <fnref n="2" />,<fnref n="3" />, Judith Apsley <fnref n="1" /> ID, Siniša Djurović <fnref n="1" />,* ID, Xiandong Ma <fnref n="2" /> ID
and Mohamed Benbouzid <fnref n="4" /> ID

**Abstract:** This paper reviews renewable energy integration with the electrical power grid through the use of advanced solutions at the device and system level, using smart operation with better utilisation of design margins and power flow optimisation with machine learning. This paper first highlights the significance of credible temperature measurements for devices with advanced power flow management, particularly the use of advanced fibre optic sensing technology. The potential to expand renewable energy generation capacity, particularly of existing wind farms, by exploiting thermal design margins is then explored. Dynamic and adaptive optimal power flow models are subsequently reviewed for optimisation of resource utilisation and minimisation of operational risks. This paper suggests that system-level automation of these processes could improve power capacity exploitation and network stability economically and environmentally. Further research is needed to achieve these goals.

**Keywords:** renewable integration; advanced solutions; thermal margin; fibre optic sensor; power flow; optimisation; machine learning

## 1. Introduction

Considerable efforts are being made to de-carbonise electrical power networks, where renewable energy resources such as wind and solar present a viable alternative to carbon-based sources. The ongoing availability and security of global energy is one of the key blockages to future sustainability [1], and further research and investment are needed for effective large-scale adoption in the coming years. The continued growth of renewables capacity, led by wind and solar, complicates the power grid composition and, in particular, how it is operated to deliver energy reliably. Intelligent solutions are needed to ensure optimal exploitation and grid integration of renewables. This paper addresses two vital aspects of renewables integration by exploring possibilities for advanced solutions in this space from both the generating device and the power system operation perspectives. The first aspect examines how the capacity of existing wind turbine (WT) generators may be expanded at low cost through advanced control to exploit device design margins. The second aspect considers the application of machine learning methods to assist with the necessary power flow optimisation in a power network dominated by low-carbon renewables.

Enhanced utilisation of the existing wind turbine capacity essentially looks at a low-cost retrofittable extension of the wind generator's nominal operational envelope. Such a solution could increase in-service capacity above the pre-installation design rating without

<figure><img src="image_2.png" /></figure>

### check for updates

**Citation:** Ghafoor, A.; Aldahmashi, J.; Apsley, J.; Djurović, S.; Ma, X.; Benbouzid, M. Intelligent Integration of Renewable Energy Resources
**Review:** Generation and Grid Level Opportunities and Challenges.
*Energies* **2024**, *17*, 4399. https://doi.org/10.3390/en17174399

Academic Editor: Ahmed Abu-Siada

Received: 1 July 2024

Revised: 20 August 2024

Accepted: 29 August 2024

Published: 2 September 2024

<figure><img src="image_3.png" /></figure>

**Copyright:** © 2024 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

<footnote n="1">¹ Department of Electrical and Electronic Engineering, School of Engineering, The University of Manchester, Manchester M13 9PL, UK; aras.ghafoor@manchester.ac.uk (A.G.); j.apsley@manchester.ac.uk (J.A.)</footnote>

<footnote n="2">² School of Engineering, Lancaster University, Lancaster LA1 4YW, UK; j.aldahmashi@lancaster.ac.uk (J.A.); xiandong.ma@lancaster.ac.uk (X.M.)</footnote>

<footnote n="3">³ Department of Electrical Engineering, College of Engineering, Northern Border University, Arar 73213, Saudi Arabia</footnote>

<footnote n="4">⁴ Institut de Recherche Dupuy de Lôme (UMR CNRS 6027), University of Brest, 29238 Brest, France; mohamed.benbouzid@univ-brest.fr</footnote>

<footnote>* Correspondence: sinisa.durovic@manchester.ac.uk</footnote>
replacing major system components. However, this requires the system components to be operated at a higher than nominal rating; hence, a better understanding of the in-service stresses is required. Improved sensing through advanced condition monitoring techniques for thermal feedback, integrated with the WT control for power generation management, is generally needed to facilitate such schemes. Thermal margins in WT electrical generators [2] and power electronic converters [3] can be sizeable, and their full exploitation could provide an increase in in-service system capacity.

The WT industry has reported limited exploration of technologies to increase the performance of in-service WT generators and hence annual energy production: up to a 5% increase is identified through the use of 'over-rating control', depending on the size and specifications of the upgraded WT system [4,5]. This was accomplished by inserting additional hardware and software upgrades, taking into account site conditions such as ambient temperature, wind speed, generator, and grid side voltages, and most importantly, the drive-train components' current loading; these are seen as key factors in determining the WT operational envelope [4]. However, ref. [6] argues that the limiting factor is hotspot temperature rather than current, so credible real-time WT drive-train temperature measurements are necessary to extend the WT operation envelope. The existing WT drive-trains employ conventional temperature sensors such as thermocouples and resistance temperature detectors for this purpose [7]. Despite offering effective, low-cost sensing solutions, these conventional sensors have access limitations and are electrically conductive, which may cause safety issues [8]. Crucially, it is difficult to locate conventional sensors where the key device hotspots occur (e.g., generator winding coil centres, power electronic switch junctions).

Fibre optic Bragg grating (FBG) sensing technology has recently emerged as a viable alternative, offering the capability for in situ, in-service distributed hotspot measurement that simultaneously provides electrical isolation and is immune to electromagnetic interference [9]. Despite its wide commercial usage for WT blade strain monitoring [10], FBG sensing applications for WT drive-train temperature measurements have not yet received much attention among WT manufacturers. However, academic research works have shown the feasibility and robustness of FBG sensors for temperature measurements in various parts of electrical machines, such as end-windings, stator slot centres, and rotor surfaces, [9,11] but also in power electronic switches where direct thermal on-chip thermal sensing was shown to be possible [12–14]. These sensing applications allow for unparalleled awareness of the thermal conditions of key device locations and could be integrated into modern WT generators, which can directly be translated into a much-improved understanding of the in-service operating envelope.

The real-time integration of temperature sensors with an electrical machine and power converter controller would thus provide a way to extend the wind generator's operating capacity in-service, past the conservative nominal values, in a controllable manner. Research works have demonstrated that bespoke electrical machine drives with closed-loop thermal feedback are integrated with the relevant field-oriented controllers for improved performance in automotive applications [15–17], but schemes of this type have not been widely researched in wind power generation. Similarly, the research on FBG sensing applications in electrical machines and drives has, to date, been largely devoted to understanding the sensing implementation without integrating these capable sensors with real-time control for improved performance management. This paper aims to review the available literature and build on this to explore a possible framework to implement FBG sensing and thermal management of a WT generator with overrating control, as well as the general requirements for its implementation.

The second aspect of this paper centres on the transformative impact that machine learning (ML) technologies have on optimal power flow (OPF) within modern power systems, which are integrating renewable energy sources at an unprecedented rate. As the energy landscape shifts towards renewables like wind and solar, the inherent variability and unpredictability of these sources pose significant challenges to traditional OPF models [18].
These models, originally designed for more stable and predictable energy sources, are not equipped to handle the dynamic fluctuations that renewable energies introduce. This situation necessitates a paradigm shift from static and deterministic OPF models to those that are dynamic and adaptive, capable of real-time analysis and response. ML offers an innovative solution, employing sophisticated algorithms to process continuous streams of data from grid sensors and smart meters. Thus, ML enables the real-time optimisation of power flows and predictive monitoring of the system's operational health. This dynamic learning and adaptive response capability ensure that the grid can maintain stability and efficiency even under the fluctuating conditions that renewables introduce [19]. Moreover, the integration of ML into OPF can lead to more informed and proactive management strategies, enhancing the grid's ability to cope with immediate and future challenges while optimising resource utilisation and minimising operational risks.

Expanding further, the incorporation of machine learning into OPF redefines the boundaries of grid management from a computational task to a strategic governance framework. With ML, the grid is not only a network of physical power flows but also a platform for intelligent decision-making, where data-driven insights lead to better control and optimisation decisions [20]. This advanced approach facilitates a transition from reactive to proactive grid management, where potential issues can be anticipated and mitigated before they escalate. Furthermore, the ability of ML to integrate with existing grid infrastructure introduces a layer of resilience and adaptability previously unattainable with conventional OPF methods [21]. This paper, therefore, also reviews the specific ML techniques that enhance OPF, such as deep learning and reinforcement learning, examining their roles in optimising grid operations against the backdrop of increasing renewable integration. This discussion complements this review by providing an outline of the necessary technological advancements and proposing changes in regulatory frameworks to effectively incorporate these intelligent systems into everyday grid operations. The underlying aim is to provide insights into a possible path forward for energy systems, emphasising the critical role of machine learning in ensuring that the grid not only survives but continues to improve its functionality in the face of evolving global energy demands and the push toward sustainability.

This paper is organised as follows. Section 2 starts with a brief background regarding the advanced monitoring and control for the over-rating operation of a WT. The rest of Section 2 is devoted to reviewing the relevant topics such as the WT thermal condition monitoring, electrical machine thermal design limitations and margins, thermal feedback integration electrical machine controller, and WT power curve upgrade. Section 3 reviews the application of machine learning methods to optimal power flow, discussing both deterministic and probabilistic OPF models, the integration of deep learning and reinforcement learning techniques, and the role of these technologies in enhancing real-time grid operation and management.

## 2. Advanced Monitoring and Control for Optimised Exploitation

This section presents a review of challenges and opportunities for improving the exploitation of WT generators through over-rating control, underpinned by advanced in situ thermal monitoring. The WT generator thermal monitoring is first reviewed, followed by an overview of the generator's operating margins and limitations. The possible control architectures are then presented, and the general implementation requirements of the thermally controlled over-rating capability in variable-speed WT generators are explored.

Monitoring of thermal, mechanical, and electrical operating parameters in wind turbines (WTs) plays a vital role in managing their in-service utilisation. This is particularly relevant for the WT drivetrain and its generator and converter, which are the main electromechanical energy conversion components. In-service abnormalities can cause deviations from recognised parameter values for operation in the nominal range [22]. The ability to measure these key operating parameters of WT subassemblies whilst in-
service is imperative for the recognition of abnormal operating states in time to establish mitigating actions.

Targeted observation of temperature rise in WT components has been used for fault detection [23]. The nominal current ratings of the WT generator and power converter are directly associated with permissible thermal levels in their windings and power electronic switch junctions, respectively. The accurate observation of worst-case, in-service temperature in these may permit over-rating power to be extracted by the WT in order to yield a desirable increase in energy recovery. The aim is to load the generator beyond the design temperature in nominal conditions whilst ensuring its integrity is not or is minimally compromised. For this to be achieved, in addition to improved monitoring, advanced control routines are needed that can react intelligently to improved sensing feedback and are able to deliver improved WT operational capability, while keeping its assets within safe integrity margins. Examples include allowing controlled overloads under cold ambient conditions or for short durations, with no or minimal risk of damage, or alternatively extending service life in faulty conditions through redistributing load to other WTs. The availability of such solutions would open attractive opportunities to develop more resilient WT systems needed to underpin our Net Zero transition.

## 2.1. Wind Turbine Thermal Condition Monitoring

WT thermal monitoring has long been used and remains standard in practical applications, with a range of thermal sensors fitted to WT drivetrains [24]. The sensor type, location, and its measurand fidelity and resolution can vary across different possible monitoring solutions, extending from, e.g., low-resolution measurements provided through WT supervisory control and data acquisition (SCADA) systems to higher resolution measurements from dedicated condition monitoring platforms [25]. This section provides an overview of the general thermal monitoring techniques and their use in WT drivetrains and identifies potential techniques for achieving improved sensing.

Existing regulations for WT system certification stipulate the minimal set of thermal and other sensing points for the entire WT structure and, in particular, its drivetrain [7]. Where thermal monitoring of the drivetrain and the generator is concerned, the use of conventional thermal sensing elements (e.g., thermocouple (TC) or resistance temperature detector (RTD)) is recommended. Sensors may be embedded in various locations of interest, such as the end winding, winding slot centre, and stator pack laminations [26]. A Siemens WT commercial condition monitoring system known as SIPLUS CM [27] utilises vibration signals as well as temperature signals measured from the WT drive train components, including the generator, through a SIMATIC S7 module supporting the use of various different TC and RTD sensors.

Conventional TC and RTD thermal sensors are electrically conductive and require wiring, so they cannot easily be placed in close contact with the active current-carrying copper conductor in an arbitrary position. Due to these sensors' installation requirements and bulk, the locations where the hottest temperatures occur can be impractical or challenging to measure; hence, the hottest temperature measurement of the active copper conductors in a machine may be underestimated. One such scenario is illustrated in Figure 1, where for practical reasons, TC sensors were installed away from the slot centre where the hotspot temperature occurs [26].

The sensors and wiring can occupy a relatively large space, so they cannot provide sufficient temperature measurement points for detailed thermal mapping. The electrical conductivity of conventional sensor-based thermal sensing makes the monitoring system complicated and less reliable [28]. In addition, conventional sensors have low immunity to electromagnetic interference (EMI).

Due to the disadvantages of TC and RTD sensors for temperature monitoring, there is a continued interest in alternative temperature sensors, which could operate effectively in EMI-rich and electrically conductive environments. A fibre optic sensing technology known as fibre Bragg grating sensors (FBGs) has emerged that offers the desired features needed
to provide improved in-service monitoring solutions for electrical machines. The FBGs can perform multi-physical sensing [11], possess a multiplexing capability, are electrically non-conductive, exhibit a high level of immunity to EMI, are suitable for use in harsh environments, and are of small size and thus suited to applications where weight and size are critical [29]. An additional beneficial feature of FBGs is their superior data transmission over a long distance without any data loss [30].

<figure><img src="image_4.png" /><figcaption>**Figure 1.** Illustration of typical conventional sensor positions.</figcaption></figure>

FBGs require a laser source and interrogation unit, which is expensive compared with TC and RTD technologies. However, FBGs are widely employed by WT manufacturers as strain sensors for WT blades and for structural health monitoring. The FBGs can be installed at multiple points on the turbine blades or tower, facilitating the detection of small cracks and abnormalities in rotating blades and their structure [10]. Examples of commercially installed FBGs in Portugal, the United Kingdom, and France for strain measurement operated at various sampling rates of 25 Hz, 100 Hz, and 2 kHz and were designed and utilised to detect cracks in the blades, unbalanced turbine towers or blades, and icing thickness [31].

Despite being commercially employed for strain measurements, FBGs have not yet received sufficient attention from WT manufacturers for drive train monitoring applications, particularly for generator and converter thermal condition monitoring. Recent research has demonstrated the feasibility of thermal sensing using FBGs embedded in various locations within electrical machines, such as stator end-windings [32], slot centres [8], rotors [33], and bearings [34], as well as power electronic switches [13]. In these studies, a single FBG or an array of FBGs was installed in the points of interest in the studied device geometry, and in-service tests were performed under different practical healthy and fault conditions, indicating reliable response and measurement of temperature.

There are specific requirements for the successful implementation of FBGs during installation and operation [35]. The FBG's intrinsic cross-sensitivity to temperature and strain needs to be addressed through appropriate sensor packaging to allow exclusive sensing of thermal only or strain only [8]. FBGs are of small size and flexible and thus allow for in situ observation of localised excitation; however, the accurate determination of precise locations of highest excitation to sense can be a challenge in practical device geometries. Experimentally verified modelling [8] has therefore been employed to determine the optimal FBG sensor positions. The FBG sensor-to-measurand interface also requires careful consideration, where suitable packaging is often required to protect the sensor and ensure proper functionality during electrical machine operation [32]. While FBGs have shown reliable, in situ thermal and other monitoring, the interpretation of the diagnostic information contained in the high-fidelity thermal data requires further research [9], including for WT drivetrains.
## 2.2. Thermal Design Limits and Margins

Three factors, electromagnetic, mechanical, and thermal, limit the current or torque density in electrical machines [36]. The saturation level of the core magnetic materials selected in the design phase determines electromagnetic limitations. The maximum mechanical operating speed is constrained by the stiffness of the bearings and shaft. The thermal limit of electrical motors and generators is determined by the winding insulation temperature, which is one of the most vulnerable parts of the machine when subjected to thermal excitation caused by nominal or abnormal operating conditions. In permanent magnet machines, temperature dependency of the demagnetisation characteristic is also a constraint.

The National Electrical Manufacturers Association (NEMA) [2] classified insulation system classes by letters A, B, F, and H, specifying thermal ratings associated with each class. The ambient temperature of 40 °C has been established as a reference for all of the insulation classes, followed by the maximum temperature rise of each class. The combination of the ambient temperature and the temperature rise determines the maximum allowed operating temperature for a given insulation class. For example, for all induction machines rated above 1 kW, continuously operating at a service factor (SF) of 1 and 1.15, insulation class A has the lowest permitted temperature rise of 60 °C and 65 °C, respectively, while insulation class H has the highest temperature rise of 125 °C and 135 °C, as shown in Table 1. The Table 1 data are obtained by the average winding temperature measurement using the "resistance method" detailed by the IEEE Std 112 [37] since winding resistance is temperature-dependent. This method neglects winding hotspot temperature measurement. To overcome this issue, NEMA utilises slot-embedded TCs and RTDs temperature sensors to measure the winding hotspot temperature in the slots. Table 2 shows the NEMA stipulated temperature rise of all insulation classes for induction machine ratings above 1120 kW at SF 1 and 1.15 (continuous operation) measured by the winding slot-embedded sensors. The limitations of slot-embedded TCs and RTDs have been detailed in the previous section: due to the practical challenges of measuring the point of highest temperature reliably with these sensors, a hot spot temperature allowance is often introduced to provide a thermal safety margin. An interesting in situ sensing alternative is presented by the FBG sensor, where sensors can be embedded in slot centre to facilitate credible measurement of the winding temperature hotspots without safety and size concerns [8].

<table><caption>**Table 1.** Insulation class rating measured by resistance method at service factors 1 and 1.15 for all induction machines above 1 kW rating [2].</caption><thead><tr><th>NEMA Insulation Class Rating<br>Measured by Resistance Method</th><th>Temperature Rise in Degrees, °C Starting from the Ambient<br>Temperature of 40 °C</th></tr></thead><tbody><tr><td>Insulation class</td><td>SF 1</td></tr><tr><td>A</td><td>60</td></tr><tr><td>B</td><td>80</td></tr><tr><td>F</td><td>105</td></tr><tr><td>H</td><td>125</td></tr></tbody></table><table><tbody><tr><td>SF 1</td><td>65</td></tr><tr><td>85</td><td>110</td></tr><tr><td>135</td><td>145</td></tr></tbody></table>

<table><caption>**Table 2.** Insulation class rating is measured by slot-embedded TC and RTD at service factors 1 and 1.15 for induction machines of over 1120 kW rating [2].</caption><thead><tr><th>NEMA Insulation Class Rating<br>Measured by Slot-Embedded<br>Sensors</th><th>Temperature Rise in Degrees, °C Starting from the Ambient<br>Temperature of 40 °C</th></tr></thead><tbody><tr><td>Insulation class</td><td>SF 1</td></tr><tr><td>A</td><td>65</td></tr><tr><td>B</td><td>85</td></tr><tr><td>F</td><td>110</td></tr><tr><td>H</td><td>135</td></tr></tbody></table><table><tbody><tr><td>SF 1.15</td><td>75</td></tr><tr><td>95</td><td>120</td></tr><tr><td>145</td><td>150</td></tr></tbody></table>
The values in Tables 1 and 2 give winding insulation temperature thresholds that are typically higher than the hotspot temperatures of in-service machines operating in their nominal rated conditions [38]. For instance, the winding hotspot temperatures of a commercial 0.55 kW induction motor and 5.5 kW permanent magnet motor, measured using FBGs in a healthy full-load continuous duty cycle (S1), were 96 °C at an ambient temperature of 23 °C [8] and 80 °C at an ambient temperature of 21 °C [32], respectively. The test motor insulations were class F with class B temperature rise, corresponding to a thermal rating of 155 °C, with an 85 °C rise, as specified by NEMA. For large machines, thermal sensing using FBGs for a 42 MW hydropower generator was reported in [39], where the recorded stator winding surface temperature was 95 °C during full-load operation conditions. Therefore, it is clear that, typically, there is a thermal design margin in practical applications. This margin offers insulation lifetime extension and further thermal safety [38], as the lifetime of winding insulation is inversely proportional to the winding operating temperature. For any 10 °C increase in winding temperature, the insulation lifetime is decreased by half [36]. Similarly, by lowering the winding operating temperature by 10 °C, the insulation lifetime is doubled. Steady-state operation within a lower temperature range can also increase thermal safety margins in overload conditions, with a variable-speed drive, and with time-varying duty cycles and transients [38]. However, the potential extra capacity that could be extracted through over-rating, by exploiting of the thermal design margins (i.e., by running windings hotter), can present attractive opportunities for increasing the output in some applications.

## 2.3. Integration of Closed-Loop Thermal Feedback with Electrical Machine Control

Despite the possible extra capacity contained in the thermal margins, only a limited number of researchers have explored the operation of electrical machines close to their thermal design limits. This would only be possible if the thermal state of the machine is reliably and accurately measured and integrated with real-time control, able to facilitate an optimal trade-off between more torque (or power) and higher temperature, which can potentially reduce insulation lifetime for a given operating scenario [40].

Closed-loop temperature feedback for active thermal management has been implemented on a switched reluctance motor [15], a permanent magnet motor [16], and an induction motor [17] for automotive applications to extract short bursts of higher manoeuvring torque. Refs. [15–17] employed model predictive control (MPC) in conjunction with simple and complex lumped parameter thermal networks for temperature estimation. Motor losses were first calculated as inputs to the thermal network, and then the temperatures were predicted, converted into a current limit, and fed back to a field-oriented torque controller, as illustrated in Figure 2. This mechanism enables a thermally controlled machine to limit the operating temperature to a desired reference point, which cannot be guaranteed in a controlled machine without thermal estimated temperature feedback either sensed or derived from a lumped parameter thermal network (LPTN) or an alternative estimator [16]. With this proposed active thermal control, if a measured temperature is lower than its set point, the machine can be pushed to allow a higher current and so torque, and if a machine’s temperature is close to/or exceeding the design limit, the controller acts to reduce the current or torque limit leading to temperature reduction. Further research explored an increase in performance of an emulated automotive drive using active thermal management integrating both the power electronic device and the motor winding temperatures in real-time with a field-oriented controller considering not only conventional voltage and current boundaries but also the thermal design limits [40]. The thermal monitoring in this work is, however, either estimated using a simple thermal model which could underestimate the temperature, or via a complex thermal model that could be subject to error and increases computational requirements. While limited, the existing research on active thermal control of electric motors and drives indicates a strong potential for intelligent and reasonably low-cost output capacity improvement. In addition, the existing work is largely based on utilising estimator type models for thermal monitoring; hence, improving the
quality of real-time thermal measurement feedback would be of benefit to further improve
the efficacy of schemes of this type in various applications.

<figure><img src="image_5.png" /><figcaption>**Figure 2.** Induction motor active thermal management using model predictive control [17].</figcaption></figure>

2.4. *Wind Turbine Overload Capability and Extracting More Energy*

Improvements in the existing WT systems to capture more wind energy through over-rating have been investigated independently by WT manufacturers. The “Energy thrust” by Siemens Gamesa (Hamburg, Germany) [4] and “PowerPlus” by Vestas (Aarhus, Denmark) [5] both claim to enable an annual increased energy production (AEP) of up to 5%. Examples of the upgraded commercial turbine models are SWT 2.3, 3, and 3.6, manufactured by Siemens Gamesa, and V82-1.65MW, V90-1.8MW, and V100-1.8MW, manufactured by Vestas. Both turbine manufacturers have upgraded the entire power curve operating regions in this process: the maximum power point tracking (MPPT) region, the constant power region, and the cut-out wind speed extension. The original and the upgraded power curves reported for a typical WT by Vestas are displayed in Figure 3 [5,41]. In the MPPT region, the aerodynamics have been upgraded using vortex generators mounted on the turbine blades: this is claimed to enhance the process of lift generation on the blades and, therefore, provide a better aerodynamically performing wind turbine rotor, resulting in more power extraction from the wind stream. In the full load operating region (i.e., the constant power region), the original power curve has been uprated by making use of load margins performed by taking into account the site condition thresholds (ambient temperature, current ratings of WT system components, gusty wind level, and the magnitudes of both converter and grid side voltages), implemented through adjustment of control parameters. In the constant power region of operation, the WT operates at a new maximised capacity with no upgrade or replacement to the core components, such as the generator or power converter hardware. The turbine cut-out wind speed is also extended from 25 m/s to 30 m/s, contributing to the increase in turbine output power.

The manufacturers have indicated that the effective implementation of these tech-
niques is highly reliant on more reliable sensing of multiple WT measurands. However, due
to the lack of information available in the public domain, the details of the existing work on
WT power curve upgrade through over-rating are not fully understood. The commercial
work adjusts the current limits in key turbine power conversion components based on
ambient temperature [41] with control of the WT operating point through a combination of
reference torque and pitch control.

Since the fundamental physical constraint is temperature rather than current, thermal
design limits and temperature measurements offer a better way to set the degree of WT over-
rating. Enhanced closed-loop thermal feedback, similar to that discussed in the previous
section for electric vehicles, could be applied to a WT [6,42] For increased power yield, a
distributed FBG sensor network is proposed to monitor in situ thermal hotspots across the
WT power electronic drive and the electrical generator to be integrated with a dedicated
real-time controller (as illustrated in Figure 4). With such feedback, an appropriate control
would be able to react to prevailing wind conditions and real-time grid demand to set the generator/drive operating point to achieve different goals, such as the following examples:

(a) Operating close to or at the thermal design limit in conditions of high wind, so the WT energy yield is increased;

(b) Temporarily exceeding the thermal design limit in a controlled fashion in scenarios requiring a sudden and large power injection into the grid, for grid frequency support, or to compensate for the failure of another WT.

<figure><img src="image_6.png" /><figcaption>**Figure 3.** Vestas typical original and upgraded WT power curves reproduced from [5].</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 4.** Vision for WT drive train controllable thermal management using FBG sensors (the asterisk refers to a set reference control value).</figcaption></figure>

The availability of such an active thermal sensing scheme would have the potential to provide more resilient WT drives capable of more intelligent usage of the existing hardware capacity.
## 2.5. Discussion and Summary

With the push towards clean energy, over-rating of existing renewable generation installations is attractive, particularly in wind power, where there are plentiful opportunities to uprate existing WTs to increase the available energy output. The key to this is ensuring improved, real-time monitoring of component temperatures with more intelligent power management. FBG temperature sensors have been shown to be effective in power conversion devices and generator systems. Moreover, FBG sensors have already been applied for structural monitoring in WTs, so some of the implementation infrastructure is available in the field. The integration of improved sensor feedback with enhanced control would allow the development of more resilient WT drives, able to utilise active thermal control for increased power output or grid support at a minimal cost where there is an already existing fibre optic interrogation infrastructure (such as that used for in situ blade strain monitoring). However, while the general cost of FBG sensing is continuously reducing and FBG sensors are now largely generally comparable in cost to alternative conventional sensing, the cost of interrogator systems needed to illuminate and operate the sensing fibres remains reasonably high. While this cost can be prohibitive for condition monitoring and sensing applications of FBG technology in low-value assets, for large high-value assets such as WT systems, it is comparable to alternative commercially available high-end condition monitoring solutions [9]. Furthermore, the operational advantages and possible ancillary service potential of WT systems retro-fitted with active thermal capability would have the potential to generate extra revenue from energy production and grid support that would, over time, offset the installation cost of in situ monitoring systems. Finally, the development of alternative low-cost solutions for reliable thermal feedback based on advanced in situ sensing-based validated thermal estimators would provide alternate low-cost methods for thermal monitoring but requires further research.

This would allow for both the improvement of legacy WT equipment that has been in field operation for an extended time and the enhancement of modern WT designs. There is already demonstrable industrial interest in the development and application of these techniques; however, much further work is needed to facilitate the over-rating functionality in the field on a large scale and ensure the methodology is transparent and applicable to more modern WT designs.

## 3. Optimal Power Flow with Machine Learning

An optimal power flow (OPF) was initially proposed in 1962 by Carpentier [43]. The OPF is a non-convex, non-linear, and large-scale optimisation problem. OPF problems have been solved by the grid operator by finding the most economical generation dispatch point to meet electric demand while satisfying all the equality and inequality constraints of the network [44]. In other words, OPF assists the grid operator in controlling the power flow within the power grid without violating grid constraints. Moreover, it gives the operator useful support in the planning and operation of the grid [45].

The OPF problems can be categorised into two groups. The first group is deterministic OPF (D-OPF), and the other group is probabilistic OPF (P-OPF) [46]. D-OPF has been widely used to solve optimal flow problems. This type of OPF does not consider stochastic features, which means explicit values of the electricity demand and sustainable generation are required to deal with this type of problem. A variety of methods have been developed to solve D-OPF, e.g., evolutionary algorithm [47] and swarm intelligence [48]. However, the nonlinearity characteristics of equality constraints in the power network introduced by loads or generators make the swarm intelligence approaches unsuitable for solving OPF problems effectively. In contrast, evolutionary algorithms can be highly effective in optimising P-OPF when the solution space is adequately small or a considerable amount of time is available for the optimisation process [49].

However, electrical power systems have now become highly stochastic and uncertain, especially when distributed generators (DGs) like wind turbines and solar photovoltaics are connected in the generation process. In fact, it is difficult to use the optimisation
methods mentioned above in solving the OPF within a sufficient time, principally when
the stochastic behaviour of the DGs and uncertainty of the demand are considered [50].

3.1. *Machine Learning Methods for OPF*

Recently, driven by the growing amount of data due to using extensively smart sensors and meters in energy production and consumption, data-driven approaches with machine learning (ML) have been developed to use these data to overcome the limitation of the aforementioned methods in solving the OPF problems. ML methods provide the system the capability to automatically learn from historical data and improve its abilities without requiring an entire system identification or prior information of the environment [51]. In other words, ML methods are an efficient tool to deal with the uncertainty of the power system by generating optimisation and control decisions in real time. Therefore, ML methods are very powerful for solving OPF in real time by taking into consideration the uncertainty and stochasticity of the power system variables. ML approaches are divided into many methods, two of which are considered the most promising approaches in solving OPF in real time, namely (i) deep learning (DL) and (ii) reinforcement learning (RL) [52].

DL is a part of machine learning. In DL, computers train the models to process and learn from raw data, and that is possible by using the deep neural networks (DNNs) model. The structure of DNNs is inspired by the human brain, which is made up of multiple layers. The first layer is the input layer, whereas the last layer is the output layer, and the layers in the middle are called hidden layers. These layers consist of many processors called neurons, which are connected to each other. The input layers receive raw data from an environment, e.g., the data from power grid components, which are sent to hidden neurons through connections. The hidden neurons become activated through weighted connections, and the results are produced from the output layer. This process is called a feed-forward neural network. If the results of DNN do not match the correct results, the backpropagation algorithm is used to update these weights optimally. The loss function is the difference between the true value and the predicted value that is obtained from DNN. The DNN uses the backpropagation algorithm, e.g., gradient descent, to reduce the difference between true and predicted values. A DL method is suitable for working in high-dimensional environments [53].

RL is also a subset of machine learning, concerned with how the agent takes a sequence of actions in a dynamic and uncertain environment in order to increase the cumulative reward. RL has a number of base elements, including agents, environments, states, actions, and rewards. An agent takes some actions in an environment to maximise the rewards. An action is the group of potential moves that the agent is able to make at each state. An environment is a place where the agent can take action. A state is a situation where the agent locates itself. RL can be formulated as a Markov decision process (MDP) that consists of state space, action space, reward function, transition probability function, and discount factor.

In data-driven RL OPF methods, the agent of RL shows great capabilities to make
sequences of decisions in the absence of power grid information. Using reinforcement
learning in a power grid decision-making has significant advantages. The agent seeks to
make optimal actions for each state by interacting with grid components. RL agents do
not require any initial knowledge to make these actions on the grid. Moreover, the RL
agent can achieve many objectives through offline training and online implementation.
Lastly, the RL is easier to apply in different scenarios in real-time OPF than traditional
optimisation approaches. The reason is that a trained RL agent is able to calculate real-time
optimisation problems in a grid within several milliseconds [54]. Consequently, the RL is a
very efficient tool for solving real-time optimisation problems. However, RL does not work
appropriately in continuous state space like OPF. Furthermore, it suffers from dealing with
large-dimension data and faces various challenges related to transition function uncertainty
and inefficient data usage.
To enhance ML performance, researchers have made efforts to fill the gap by combining RL with DL to create deep reinforcement learning (DRL). As mentioned above, RL has great capabilities to make sequences of decisions in an uncertain environment by learning the optimal action through interactions with a stochastic or deterministic environment. To increase the performance of RL in solving high-dimensional real-time problems, researchers have combined a deep neural network (DNN) with RL, where the DNN works as a function approximator.

## 3.2. OPF Based on Objective Functions

Each optimisation problem, like OPF, has a dedicated objective function, which needs to be optimised with respect to the target variables of the power system in the presence of constraints imposed on those variables. The aim of this part is to classify the OPF in terms of objective function. Different DRL approaches are applied to find the best OPF solution for the proposed objective function.

### 3.2.1. Operating Cost Minimisation

OPF supports the network operators, minimising operational costs. Since reducing the electricity cost is considered one of the main goals for the operator of the grid, it has been used widely as an objective function [55].

Due to the high-level penetration of distributed generators (e.g., solar PVs, wind turbines) in distributed networks, controlling these devices becomes very important to minimise the running cost. In ref. [18], a soft actor-critic is proposed for solving the optimal active power dispatch on the IEEE 118-bus. The Lagrange multiplier method is used to improve the performance of the soft actor-critic algorithm in a high-dimensional discrete action environment. The proposed algorithm is more effective in finding active power dispatch points than the proximal policy optimisation and double deep Q-network. To deal with a continuous action space, the authors in [56] introduce a Lagrangian-based DRL to solve the continuous real-time OPF. The objective of this work is to find the least generation dispatching cost while satisfying the security constraints. The critic networks are not used because they induce higher approximation errors. Instead, the deterministic gradient is approximated analytically. The proposed method reached the best solution compared to the supervised learning method. A twin-delayed deep deterministic policy gradient (TD3) algorithm is used in [19] to minimise the summation of production costs by determining the active power of the generators on the IEEE 118-bus system, where a Levenberg-Marquardt method is introduced to the TD3 to mitigate the risk of divergence solutions. The proposed method is able to find a better solution than the deep deterministic policy gradient (DDPG) that is used in [56].

Energy storage (ES), on the other hand, is used widely in electrical grids to store excess power from distributed generators and can be managed optimally to minimise operating costs. In [57], DDPG is proposed to control a battery with lookahead constraints in real time. A safety layer and two replay buffers are introduced to promote the RL agent's action, where the goal of the agent is to increase revenue by operating the energy storage optimally. The proposed method can reach a cost that is close to the ideal cost while the computational time is reduced multiple times as compared with model predictive control (MPC). In [58], a DRL-based method is proposed to control energy storage and distributed generators in a microgrid to reduce the purchases of power from the main grid. The authors in [59] proposed a bottom-up energy internet architecture to model the integrated multi-microgrid to minimise the overall cost by the optimal control of the energy storage and distributed generators. The DRL method is utilised to manage the power sources in the bottom layer and dispatches the decision to the up layer, which is connected to the main grid. The simulation results show that the proposed method outperforms MPC in minimising the running cost.

Due to the increasing number of electric vehicles (EVs) that are able to work as a load or a power source, the authors in [60] developed a control strategy to minimise the power cost
in a microgrid by considering stochasticity associated with electricity price and renewable resources. TD3 algorithm is utilised to control the distributed generators and electric vehicles, and simulation results show that the proposed control strategy outperforms the traditional particle swarm optimisation (PSO) method. To deal with the unknown transition probability of a distribution network equipped with large-scale electric vehicle charging and distributed generators, the nodal multi-target policy is proposed in [61] to schedule the optimal electric vehicle charging while a soft actor-critic algorithm is used to determine the target levels for the policy. The proposed approach achieves lower system costs than the proximal policy optimisation (PPO) method.

Flexible loads are considered one of the most efficient ways to minimise operating costs. To study the feasibility of using flexible loads, the authors in [62] proposed graph reinforcement learning to manage an electrical network that contains both energy storage and flexible loads. The proposed method is implemented based on a graph attention network to extract the topology structure information from the electrical grid and send this information to DDPG to find the optimal formulation in order to manage the controllable assets. The proposed method is carried out within an IEEE 123-bus system, and the simulation results show the ability of the method to find the optimal operational status compared to PSO. To exploit the interruptible loads at the demand side, the authors in [63] used the duelling deep Q network (DQN) algorithm to minimise the daily load cost.

When faults occur in the distributed networks, the grid operators often try to disconnect a number of buses to isolate the affected transmission lines, attempting to ensure the grid works continuously without considering the operational cost. The authors in [64] proposed a method to minimise the running cost even when the faults occur by optimal controlling the topology and distributed generators. Three-stage reinforcement learning is presented to manage an IEEE 33-bus system, and the simulation results show the capability of this approach to reduce operating costs even when one of the transmission lines is disconnected. In [65], a batch-constrained soft actor-critic algorithm is developed to minimise the operational cost by finding the optimal configuration for a power grid under unforeseen states. The test results show that the proposed method is better than DQN and SAC in terms of decreasing the system running costs.

Minimising power loss is deemed one of the techniques to reduce the overall operating cost by controlling the active and reactive power of the controllable component in the electric grid. In [55], the OPF is modelled as a stochastic nonlinear programming problem, and the proximal policy optimisation (PPO) is proposed to find the best solution for the optimisation problem by modifying the active and reactive power of the energy storage. The DRL-based approach reaches the least operational cost for IEEE 33-bus compared to stochastic programming. In [66], TD3 is presented to optimally control the community microgrid networks with integrated solar PVs, wind turbines, and energy storage. The DRL agent is able to manage the active and reactive power of the grid to minimise the total power loss. The related work for minimising the operating cost is summarised in Table 3.

<table><caption>**Table 3.** OPF through minimising the operating cost.</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[18]</td><td>Lagrange multiplier SAC</td><td>Maximise operational rewards, minimise generation costs, maintain system constraints (power balance, voltage limits)</td><td>Optimal active power dispatch for DGs in systems with renewable energy</td><td>Discrete actions</td><td>IEEE 118-bus power system</td></tr><tr><td>[56]</td><td>Lagrangian-based DDPG</td><td>Minimise total generation cost</td><td>Optimising real-time power flow in systems with intermittent distributed renewable generators</td><td>Continuous action</td><td>IEEE 118-bus power system</td></tr></tbody></table>
<table><caption>**Table 3. Cont.**</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[19]</td><td>TD3 and Levenberg-Marquardt</td><td>Minimise total generation cost</td><td>Real-time optimal power flow management, controlling DG outputs</td><td>Continuous action</td><td>IEEE 118-bus power system</td></tr><tr><td>[57]</td><td>DDPG with safety layer and dual replay buffer</td><td>Minimise expected time average cost over control horizon, accounting for monetary costs at PCC and dispatch plan deviation cost</td><td>Real-time control of ES in active distribution grids with distributed energy resources</td><td>Continuous action</td><td>34-bus Swiss grid</td></tr><tr><td>[58]</td><td>DQN</td><td>Minimise operational costs of microgrid</td><td>Management of conventional generators, DGs, ESs, and grid interactions</td><td>Discrete actions</td><td>Microgrid (size not specified)</td></tr><tr><td>[59]</td><td>A2C with Curriculum Learning</td><td>Minimise overall operational costs of microgrids, considering generation costs, storage management, and penalties for operational constraint violations.</td><td>Energy management of controllable DGs in a network of microgrids</td><td>Continuous action</td><td>Network of interconnected microgrids (size not specified)</td></tr><tr><td>[60]</td><td>TD3</td><td>Minimise operating costs, including generation, transaction, and EV charging costs, with incentives for renewable energy use</td><td>Intelligent energy management in a hybrid DG and EV system</td><td>Continuous action</td><td>Microgrid (size not specified)</td></tr><tr><td>[61]</td><td>SAC with nodal multi-target (NMT) approach</td><td>Minimise energy costs and penalties related to EV charging non-completion</td><td>EV charging scheduling in a power distribution network</td><td>Continuous action</td><td>IEEE 37-node test feeder with 2500 EV stations</td></tr><tr><td>[62]</td><td>Graph RL with Graph Attention Networks (GAT) and DDPG</td><td>Minimise costs, including network transactions, power losses, load control, and voltage deviations</td><td>Real-time optimal scheduling of FLs, DGs, ES systems, and SVCs in active distribution networks</td><td>Continuous action</td><td>Modified IEEE 33-bus system</td></tr><tr><td>[63]</td><td>DDQN</td><td>Maximise long-term profit by managing interruptible loads to reduce peak demand and operation costs while maintaining voltage limits</td><td>Demand response management of interruptible load (FL) in power distribution networks</td><td>Discrete actions</td><td>Enhanced IEEE 33-node test feeder system</td></tr><tr><td>[64]</td><td>Three-stage DDQN and DDPG</td><td>Minimise operational costs while ensuring network stability and reliability, reducing power losses, managing load demands, and optimising distributed energy resources (DERs)</td><td>Real-time operation of distribution networks, controlling DGs, load points, switches, and ESSs</td><td>Discrete and continuous actions</td><td>IEEE 33-bus system</td></tr></tbody></table>
<table><caption>**Table 3.** *Cont.*</caption><thead><tr><th>Ref.</th><th>Optimisation Method</th><th>Objective Function</th><th>Application</th><th>Action</th><th>Power System Size</th></tr></thead><tbody><tr><td>[65]</td><td>Batch-constrained soft actor-critic (BCSAC)</td><td>Minimise overall operational costs, including electricity consumption, line losses, and switching operations</td><td>Dynamic distribution network reconfiguration, controlling remotely operable switches</td><td>Discrete actions</td><td>119-bus distribution network</td></tr><tr><td>[20]</td><td>PPO</td><td>Minimise the cost of power loss across the distribution network with constraints related to renewable energy and storage device operations</td><td>Optimal power flow in networks with DGs and ESs</td><td>Continuous actions</td><td>Modified IEEE 33-bus network with added renewables and storage</td></tr><tr><td>[66]</td><td>TD3</td><td>Minimise total power loss within a community microgrid</td><td>Power flow optimisation in community microgrids, controlling DERs and ESS</td><td>Continuous actions</td><td>IEEE 14-bus test system</td></tr><tr><td>[67]</td><td>Improved DRL</td><td>Minimise operational costs in day-ahead dispatch, including costs from power purchases, network losses, and curtailment of renewable energy</td><td>Day-ahead optimal dispatch in active distribution networks, controlling DGs, ESS, and CBs</td><td>Discrete and continuous actions</td><td>IEEE 33-bus test system</td></tr><tr><td>[68]</td><td>Deep LSTM-based DQN</td><td>Minimise overall daily operational cost of the grid-tied microgrid, optimising power flow from BESS and managing grid interactions to reduce costs and maximise revenue</td><td>Economic energy dispatch of ES systems in a grid-tied microgrid</td><td>Discrete actions</td><td>Residential microgrid (specific size not provided)</td></tr><tr><td>[69]</td><td>Multi-agent deep deterministic policy gradient (MADDPG)</td><td>Minimise total generation and interaction costs, balancing production costs and revenue/costs associated with power flow between microgrids and the main grid</td><td>Economic dispatch of ES and DG in active distribution networks with multiple microgrids</td><td>Continuous actions</td><td>Multiple microgrids, unspecified the size</td></tr><tr><td>[70]</td><td>Double DQN</td><td>Minimise reactive power-related losses and voltage deviations, expressed as a weighted sum of costs associated with line losses and voltage deviations</td><td>Reactive power optimisation in distribution networks, controlling reactive power compensators</td><td>Discrete actions</td><td>IEEE 37-bus test system</td></tr><tr><td>[71]</td><td>DQN</td><td>Minimise annual operational costs, including energy losses and the operation of dispersed generation units</td><td>Energy management in distribution networks with EVs and DGs</td><td>Discrete actions</td><td>57-bus IEEE grid</td></tr></tbody></table>

### 3.2.2. Voltage Deviation Minimisation

An increased number of distributed generators in an electrical grid may lead to a disturbance in the voltage of the grid. The high penetration of these resources could cause
unforeseen fluctuations in the voltage profile due to their stochasticity nature [72]. Ineffective control of the grid voltage affects the power flow dispatch in the distribution networks; therefore, transmission line losses and electrical prices will eventually increase [73].

One of the techniques to improve the voltage quality is to control the distributed generators in an effective way. Optimal reactive power control of the distributed generators is used widely to decrease the fluctuation of the voltage. The authors in [74] used DDPG to control the reactive power of the PV inverters in a low-voltage network. Their simulation results show that the proposed method is able to keep the voltage fluctuation within the desired limits. The MADDPG algorithm and the attention model are used in [75] to enhance the voltage control strategy in the IEEE-123-bus system, where the results demonstrated that the proposed approach can achieve a better control performance as compared with a standard MADDPG algorithm. The authors in [76] proposed a two-stage control scheme to manage DG inverters in the IEEE 123-bus system. In the first stage, which is called an offline stage, a jointly adversarial soft actor-critic algorithm is used to make the inverter agents more robust to reach an optimal solution. Then, the SAC is used in the second stage (online stage) to control the inverters in real time. The proposed method outperforms the state-of-art DRL algorithm. Instead of using smart inverters to control the voltage profile, the authors in [77] proposed the PPO and imitation learning method to find the optimal set-points for 38 conventional generators in Illinois 200-bus systems to ensure the voltage within the acceptable range. The proposed method was able to solve the OPF problem much faster than the interior-point method.

Energy storage technologies have experienced huge development recently; as a result, they have become another feasible solution for reducing voltage fluctuation. Energy storage can play an important role in distribution networks to participate in minimising power fluctuations caused by distributed generators [78]. DQN is proposed in [79] to mitigate voltage fluctuations by controlling a single battery. The results showed that a battery is able to reduce voltage violation caused by the stochasticity of the distributed generators. Overvoltage issues are caused by high levels of penetration of distributed generators, and energy storages may not be sufficient to ingest extra power, especially during the light load intervals. Energy storage capacity problems are addressed in [21], where reinforcement learning is combined with MPC to prevent voltage violations under high generating conditions. Electric vehicles can be considered as mobile energy storage, which can play a significant role in supporting grid voltage. The average weighted deep double Q-network (DDQN) algorithm is introduced in [80] to work as a voltage controller for EVs. The proposed method outperformed DDQN and DQN in terms of keeping the voltage within safe limits. A multi-agent DQN approach is used in [81] to control EVs and ESs in a low-voltage grid. The distributed generators are integrated with energy storage units to mitigate the risk of voltage fluctuation. The authors of [82] used DDPG to find the optimal schedule of PV and energy storage inverters in an IEEE-34 bus system, and they achieved a better performance as compared to DQN in minimising voltage fluctuations.

Another approach to regulating the voltage of the grid is to use capacitor banks, which are essentially one type of reactive power compensation device. DQN algorithm is proposed to control two capacitor banks in a microgrid [83]. Capacitor banks are categorised as slow-timescale devices based on response speed. A two-timescale voltage management plan is, therefore, developed in [84] to minimise voltage deviations. DQN algorithm is utilised to optimise the setpoints of PV inverters on a fast timescale to reduce the instantaneous voltage violations. Capacitor banks can also be configured by the proposed algorithm to control long-term voltage deviations. The second type of reactive power compensation device is associated with the on-load tap changers, which can be used to regulate the voltage. The DDPG algorithm is used in [85] to learn an optimal setting of on-load tap changers in terms of mitigation of the voltage sags. A constrained soft actor-critic algorithm is presented in [86] to find an optimal configuration of on-load tap changers and capacitor banks. The simulation results show that the proposed algorithm achieves better performance than the state-of-the-art RL algorithms and the conventional optimisation-based algorithms.
A static VAR compensator (SVC) is a compensation device used to provide fast-acting reactive power in distribution systems. The soft actor-critic algorithm is introduced in [87] to enhance the ability of the grid to accommodate the high fluctuation of the voltage caused by DGs. The proposed algorithm appears to be the best for controlling the reactive power of PV inverters and SVCs to mitigate the risk of voltage violations compared to the PSO algorithm. A multi-agent soft actor-critic algorithm is used to achieve decentralised control of SVCs and energy storage units for voltage regulation in the distribution system [88]. The sparse pseudo-Gaussian process is integrated with the proposed algorithm to learn the relationship between the power injections and voltage magnitude of each bus. The results show that the multi-agent soft actor-critic (MASAC) outperformed the single-agent SAC and the traditional optimisation-based algorithms.

Reconfiguration of the distribution network plays a significant role in increasing the voltage quality of the grid by finding the optimal configuration of switching devices over a particular time period. The DQN algorithm is used as a smart controller to manage the power flow by controlling grid switches to make the voltage fluctuation within acceptable limits [89]. To examine the ability of the network reconfiguration approach to reduce the voltage violation under different loading and generating conditions, the PPO algorithm is proposed in [90] to control nine switches (sectionalism and tie switches) in a microgrid. The experimental results show that the proposed algorithm produces an effective and much faster solution than DQN.

Load shedding is considered one of the most effective and economical approaches to protect the power system against voltage swings. The DDPG algorithm is combined with the convolutional neural networks to learn the optimal load-shedding configuration to maximise voltage stability [91]. The proposed method successfully increased the quality of the voltage by determining the location and amounts of load shedding in the New England 39-bus system. The MASAC approach is also presented in [92] for voltage regulation in a low-voltage network, where the MASAC algorithm uses a decentralised execution framework to control loads in commercial buildings for mitigating voltage swings. The experimental results demonstrate that MASAC outperformed the MADDPG algorithm. The related work for minimising the voltage deviation is summarised in Table 4.

<table><caption>**Table 4.** OPF through minimising the voltage deviation.</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[74]</td><td>DDPG</td><td>Minimise voltage fluctuations</td><td>DGs</td><td>Continuous action</td><td>IEEE 21-bus</td></tr><tr><td>[75]</td><td>MADDPG</td><td>Minimise voltage fluctuations</td><td>DGs</td><td>Continuous action</td><td>123-bus systems</td></tr><tr><td>[76]</td><td>JASAC</td><td>Minimise voltage deviation and active power loss</td><td>DGs</td><td>Continuous action</td><td>IEEE 123-bus</td></tr><tr><td>[77]</td><td>PPO with IL</td><td>Minimise voltage deviation and generator costs</td><td>Conventional generators</td><td>Continuous action</td><td>IEEE 200-bus</td></tr><tr><td>[79]</td><td>DQN</td><td>Minimising the average voltage fluctuation and maximising the SoC of energy storage</td><td>ESs</td><td>Discrete action</td><td>IEEE 33-bus</td></tr><tr><td>[21]</td><td>RL with MPC</td><td>Minimise voltage deviation, active power loss, and overall cost</td><td>ESs</td><td>Continuous action</td><td>IEEE 33-bus</td></tr><tr><td>[80]</td><td>AWDDQN</td><td>Minimise voltage fluctuations</td><td>EVs and DGs</td><td>Discrete action</td><td>IEEE 123-bus system</td></tr></tbody></table>
<table><caption>**Table 4. Cont.**</caption><thead><tr><th>Ref.</th><th>Optimisation Method</th><th>Objective Function</th><th>Application</th><th>Action</th><th>Power System Size</th></tr></thead><tbody><tr><td>[81]</td><td>MADQN</td><td>Increase the proportion of PV power generation used locally and minimise voltage fluctuations</td><td>EVs and ESs</td><td>Discrete action</td><td>Low voltage grid</td></tr><tr><td>[82]</td><td>DDPG</td><td>Voltage regulation and power loss minimisation</td><td>DGs and ESs</td><td>Continuous action</td><td>IEEE-34</td></tr><tr><td>[83]</td><td>DQN</td><td>Minimise voltage deviation</td><td>CBs</td><td>Discrete action</td><td>13-bus</td></tr><tr><td>[84]</td><td>DQN</td><td>Minimising the long and short-term average voltage deviation</td><td>DGs and CBs</td><td>Discrete action</td><td>The Southern California Edison 47-bus</td></tr><tr><td>[85]</td><td>DDPG</td><td>Minimising the voltage swell and power losses</td><td>TCs</td><td>Continuous action</td><td>IEEE 33-bus system</td></tr><tr><td>[86]</td><td>Constrained SAC</td><td>Minimising voltage deviation and generating costs</td><td>TCs and CBs</td><td>Discrete action</td><td>123-bus</td></tr><tr><td>[87]</td><td>SAC</td><td>Minimising voltage deviation and power loss and generating cost</td><td>DG inverters and CBs</td><td>Continuous action</td><td>33-bus</td></tr><tr><td>[88]</td><td>Multi-agent SAC and sparse pseudo-Gaussian process</td><td>Minimising the voltage deviation and PV curtailment</td><td>ESs, DGs, and SVCs</td><td>Continuous action</td><td>IEEE 123-bus</td></tr><tr><td>[93]</td><td>Multi-agent RL algorithms</td><td>Minimising voltage deviation and active power loss</td><td>Conventional generators and capacitor bank</td><td>Discrete action</td><td>IEEE 162-bus</td></tr><tr><td>[94]</td><td>DDPG</td><td>Minimising voltage deviation and active power loss</td><td>TCs</td><td>Continuous action</td><td>IEEE 123-bus</td></tr><tr><td>[95]</td><td>DQN</td><td>Minimising the voltage deviation</td><td>Conventional generators, CBs, and TCs</td><td>Discrete action</td><td>IEEE 14-bus</td></tr><tr><td>[96]</td><td>Multi-agent SAC</td><td>Minimising the voltage deviation</td><td>DGs, CBs, and TCs</td><td>Continuous action</td><td>IEEE 123-bus</td></tr><tr><td>[97]</td><td>DDPG and Monte Carlo</td><td>Minimising voltage deviation and power loss</td><td>EVs, CBs, and TCs</td><td>Continuous action</td><td>IEEE 123-bus</td></tr><tr><td>[89]</td><td>DQN</td><td>Minimising voltage deviation, power loss and switch action cost</td><td>Topology</td><td>Discrete action</td><td>Taiwan power company 84-bus</td></tr><tr><td>[90]</td><td>PPO</td><td>Minimising voltage deviation and power loss</td><td>Topology</td><td>Discrete action</td><td>IEEE 34-bus</td></tr><tr><td>[91]</td><td>DDPG with CNN</td><td>Minimising voltage deviation</td><td>Load shedding (FL)</td><td>Continuous action</td><td>New England 39-bus system</td></tr></tbody></table>
<table><caption>**Table 4.** *Cont.*</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[92]</td><td>Multi-agent SAC</td><td>Minimising voltage deviation, energy cost and indoor thermal discomfort</td><td>Load shedding (FL)</td><td>Continuous action</td><td>Low-voltage network 6-bus</td></tr><tr><td>[98]</td><td>Convolutional LSTM with DQN</td><td>Minimising short-term voltage deviation</td><td>Load shedding (FL)</td><td>Discrete action</td><td>China Southern Power Grid 23-bus</td></tr></tbody></table>

### 3.2.3. Emission Cost Minimisation

Climate change and global warming are considered to be among the main challenges facing our world presently. Traditional generators and vehicles produce almost 60% of greenhouse gases [98]. The reason behind using fossil fuel-based generators is their low prices and reliability in contrast to DGs. Greenhouse gas emissions must be reduced to save our planet. Due to the increasing environmental awareness, the DGs have grown unprecedentedly in distribution systems; however, this growth creates significant problems for the grid. Power curtailment of DGs is necessary to minimise the voltage rise and congestion.

Real-time OPF can play an important role in minimising the curtailment of renewable energy and maximising the quality of the voltage and grid capacity. Management of the DG outputs is one approach to increase the penetration of renewable energy sources.

The authors in [99] proposed a duelling DQN algorithm to control DGs in an IEEE 14-bus system fed with 40% renewable energy sources. The experimental results showed that duelling DQN is able to increase the capacity of the grid to accommodate higher renewable energy rates and maintain the stability of the system. A two-timescale control framework is presented in [100] to manage a grid with high PV penetration (120% of the feeder capacity). In a slow timescale control, a model-based approach is used to organise the voltage. On the other hand, the DDPG algorithm is proposed to control the setpoints of PV inverters in a fast timescale. The results showed that the proposed framework achieved lower voltage deviations and PV curtailment in contrast to a traditional optimisation method based on Volt-VAR control. A soft actor-critic-based multi-agent DRL algorithm is proposed in [101] to control the active and reactive PV inverter for the Colorado U.S. grid with 80% penetration of renewable power. The proposed method succeeded in managing 77 PVs in the 759-bus system and minimised PV curtailment while keeping the grid voltage within acceptable limits as compared with the traditional Volt-VAR control method.

Energy storage units and EVs are effective methods to decrease the curtailment of DGs by storing excess power, especially when the demand is low. A microgrid with hydrogen storage units is introduced in [102], and the DDPG algorithm is used as a control agent to reduce the curtailment of PV generation. Simulation results showed that the DRL agent reduced the operation and emission cost by 5% when compared with the genetic algorithm. A vehicle-to-grid framework is developed in [103] to utilise EV features to support the grid. This framework helps the EVs to work in a cooperative way to achieve a number of goals, e.g., minimising the operational and emission costs. A hybrid multi-agent PPO algorithm is used to determine the routing and scheduling of the EVs inside the grid. Moreover, the parameter-sharing method is integrated with DRL to stabilise the training performance. The results showed that the proposed framework is able to reduce the travelling time of the EVs, as well as energy and emission costs.

The optimal configuration of the status of switches in a distribution network can play a vital role in raising the hosting capacity of transmission lines, and the curtailment of the excess renewable energy is, therefore, minimised. The DQN algorithm is used in [104] to find the best network configuration for a 16-bus distribution system. Simulation results showed that the proposed policy is able to minimise the operating cost and the curtailment power of DGs while the voltage profile is improved. Optimal management of reactive power devices can also be used to increase the hosting capacity of DGs. The multi-agent
DRL algorithm is investigated in [105] to control the bus voltages by specifying the setpoint for the SVCs. The MADDPG agents succeeded in decreasing the system loss and improving the hosting capacity of the grid as compared with a conventional model-based method. The related work for minimising the emission cost is summarised in Table 5.

<table><caption>**Table 5.** OPF through minimising the emission cost.</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[100]</td><td>Duelling DQN</td><td>Minimising operating costs and curtailment of RE</td><td>DGs</td><td>Discrete actions</td><td>IEEE 14-bus</td></tr><tr><td>[101]</td><td>DDPG and model-based approach</td><td>Minimising line losses, voltage deviations, and curtailment of RE</td><td>DGs</td><td>Continuous action</td><td>IEEE 34-bus</td></tr><tr><td>[102]</td><td>MASAC</td><td>Minimising voltage deviations and curtailment of RE</td><td>DGs</td><td>Continuous action</td><td>Colorado U.S. grid 759-bus</td></tr><tr><td>[103]</td><td>DDPG</td><td>Minimising operating and emission cost</td><td>DGs and ES</td><td>Continuous action</td><td>Microgrid</td></tr><tr><td>[104]</td><td>MAPPO with parameter-sharing</td><td>Minimising operating and emission cost</td><td>EVs</td><td>Continuous + discrete action</td><td>15-bus radial distribution</td></tr><tr><td>[105]</td><td>DQN</td><td>Minimising line losses, voltage deviations, and curtailment of RE</td><td>Reconfiguration (Topology)</td><td>Discrete action</td><td>16-bus</td></tr><tr><td>[106]</td><td>DQN with multi-objective bacterial foraging optimisation</td><td>Minimising PV power curtailment, power loss, and generation cost</td><td>Reconfiguration (Topology)</td><td>Discrete action</td><td>IEEE 118-bus</td></tr><tr><td>[107]</td><td>MADDPG</td><td>Minimising line losses, voltage deviations, and curtailment of RE</td><td>SVC</td><td>Continuous action</td><td>IEEE 300-bus and China 157-node</td></tr></tbody></table>

### 3.2.4. Increasing System Reliability

Solving real-time OPF problems is an effective method to increase the robustness and reliability of the power system to withstand any type of contingencies without violating system constraints. The authors in [108] utilised the PPO algorithm to find the optimal generator setpoints in a 200-bus system. One random transmission line outage is included to assess the performance of the proposed algorithm under a contingency state. The results demonstrated that the PPO algorithm is able to deal with topology changes and find near-optimal OPF solutions. Optimal management of PV inverters can also be used to increase the robustness of the system. The PPO algorithm is investigated in [109] to mitigate the voltage unbalance at the point of common coupling by controlling the Volt-VAR of the PV inverters.

Energy storage units and EVs can play essential roles in supporting system reliability. The DDPG algorithm is presented in [110] to reduce power fluctuations caused by large wind fluctuations. The proposed method is able to efficiently manage the energy storage units to minimise wind fluctuation as compared with the DQN algorithm. Energy storage units can be used as an efficient approach for peak load shifting. The DDPG algorithm is used in [111] to deal optimally with the uncertainty of load demand at peak time by controlling the storage units. Q-learning algorithm has been used in grid-to-vehicle and vehicle-to-grid services to increase the efficiency of the grid by minimising the peak load [112]. The authors in [113] used energy storage units controlled by the SAC algorithm to reduce the voltage violations in an unbalanced low-voltage grid.
Short-term voltage instability is a fast event that usually takes seconds, where fast actions are required to return the voltage to the normal range. Load shedding is one of the effective emergency methods to deal with voltage instability, especially short-term events. A parallel augment random search (PARS) algorithm is adopted in [114] to mitigate short-term voltage by shedding 20% of the total load. The DRL algorithm is integrated with LSTM to support the learning rate. The proposed algorithm outperformed the MPC approach in terms of computational efficiency and robustness in learning. The load-shedding method is often used in an emergency state. The DDPG algorithm is proposed in [115] to deal with line faults by using the load-shedding method, where the DRL approach is used to choose which bus participates in the shedding process and the amount of load shedding. The shedding must be less than 40% of the original load power. The results showed that the proposed approach is able to return the voltage of the bus to the normal value after the emergency event.

Topology reconfiguration is one of the best approaches for the grid operator to increase the stability of the network. Topology reconfiguration is considered the most economical solution for distribution violations when compared with other approaches like load shedding, peak shaving, and transmission line expansion. The actor-critic (A3C) algorithm is combined with domain knowledge of power system operators to prevent cascading line outages by using topology reconfiguration [116]. Due to the high generation of DGs in a distribution system, an online reconfiguration scheme is proposed in [117] to alleviate line congestion and voltage violations. The DQN algorithm is used as a controller to find optimal distribution topologies. The performance of the DRL algorithm outperformed the genetic algorithm and Brute-force Search. The related work for minimising the system instability is summarised in Table 6.

<table><caption>**Table 6.** OPF through minimising the system instability.</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[108]</td><td>PPO</td><td>Maximise power system security</td><td>Conventional generators</td><td>Continuous action</td><td>200-bus</td></tr><tr><td>[109]</td><td>PPO</td><td>Minimise voltage unbalance at the PCC</td><td>DGs</td><td>Continuous action</td><td>IEEE 34 bus</td></tr><tr><td>[110]</td><td>DDPG</td><td>Minimising the power fluctuations and power cost</td><td>ESs</td><td>Continuous action</td><td>IEEE 14 bus</td></tr><tr><td>[111]</td><td>DDPG</td><td>Minimising the operating cost and net load fluctuations</td><td>ESs</td><td>Continuous action</td><td>11 bus</td></tr><tr><td>[118]</td><td>Safety-constrained SAC</td><td>Peak shaving and voltage regulation</td><td>DG inverters and ESs</td><td>Continuous action</td><td>IEEE 123-bus</td></tr><tr><td>[112]</td><td>Q-learning algorithm and enhanced Grasshopper optimisation</td><td>Peak shaving and minimising the power loss</td><td>DG and EVs</td><td>Discrete action</td><td>IEEE 33-bus</td></tr><tr><td>[113]</td><td>SAC</td><td>Minimising the total daily cost and voltage regulation in unbalanced grid</td><td>ESs</td><td>Continuous action</td><td>IEEE 34-bus</td></tr><tr><td>[114]</td><td>PARS with LSTM</td><td>Minimising the total load-shedding amount and the voltage violations</td><td>Load shedding (FL)</td><td>Continuous action</td><td>IEEE 300-bus</td></tr><tr><td>[115]</td><td>DDPG</td><td>Minimising voltage violations</td><td>Load shedding (FL)</td><td>Continuous action</td><td>IEEE 39-bus</td></tr></tbody></table>
<table><caption>**Table 6. Cont.**</caption><thead><tr><td>Ref.</td><td>Optimisation Method</td><td>Objective Function</td><td>Application</td><td>Action</td><td>Power System Size</td></tr></thead><tbody><tr><td>[116]</td><td>A3C</td><td>Minimising the total line loading</td><td>Topology</td><td>Discrete action</td><td>IEEE 14-bus</td></tr><tr><td>[117]</td><td>DQN</td><td>Mitigating line congestion and voltage violations</td><td>Topology</td><td>Discrete action</td><td>IEEE 123-bus</td></tr><tr><td>[119]</td><td>Q-learning</td><td>Minimising the load shedding cost and frequency instability</td><td>DGs, ES, and demand response (FL)</td><td>Discrete action</td><td>IEEE 37-node MG</td></tr><tr><td>[120]</td><td>Clipped PPO</td><td>Maximising long-term voltage stability</td><td>ES and demand response (FL)</td><td>Continuous action</td><td>Nordic 32-bus</td></tr><tr><td>[121]</td><td>Q-learning with CNN</td><td>Maximising frequency stability after the fault</td><td>Load shedding (FL)</td><td>Discrete action</td><td>IEEE 39-bus</td></tr></tbody></table>

Abbreviations used in the tables are as follows: soft actor-critic: SAC; deep deterministic policy gradient: DDPG; twin delayed DDPG: TD3; duelling deep Q network: DDQN; proximal policy optimisation: PPO; multi-agent deep reinforcement learning: MADRL; distributed generators: DGs; energy storage: ES; electric vehicle: EV; flexible load: FL; static var compensation: SVC; microgrid: MG; jointly adversarial soft actor-critic algorithm: JASAC; imitation learning method: IL; weighted deep double Q-network: AWDDQN; capacitor bank: CB; tap-changer, TC.

### 3.3. Discussion and Summary

Figure 5 below reveals significant insights into the integration of DRL with OPF in terms of categories of objective function, application, optimisation method, and power system size. The use of objective functions emphasises a predominant focus on minimising costs (56%), followed by managing voltage fluctuations (26%) and emissions (11%), which underscore the economic, operational, and environmental imperatives in current research. Applications are largely dominated by DGs at 36%, highlighting a shift towards decentralised power generation models, while ES systems, EVs, and VRD (voltage regulation devices) at 20%, 5%, and 16%, respectively, appear as emerging roles in dynamic grid management. The utilisation of DQN and DDPG methods at 28% and 22% indicates a robust exploration of DRL techniques suitable for the complex and high-dimensional state spaces typical in power systems. This evolving integration from simple cost minimisation to complex objectives like volatility and emissions management reflects a maturing field where future work must address scalability and real-world applicability, ensuring advancements in ML to propel the transition toward more sustainable and resilient power systems while aligning with evolving regulatory frameworks to maximise benefits and mitigate associated risks.

Our reviews explore the significant role of ML in enhancing OPF for electrical networks that integrate renewable energy sources. OPF is crucial for determining the optimal operational points of distributed generators, with objectives such as minimising operational costs or reducing power loss. These operational points must adhere to stringent constraints, particularly maintaining network voltage within specified limits to ensure stability and efficiency. Exceeding these voltage thresholds renders the operational points unacceptable, highlighting the importance of precise control mechanisms. Through our review, we found that all studied approaches not only aim to optimise cost functions but also prioritise maintaining network voltage within acceptable limits. The integration of ML techniques has shown promising results in managing these complexities more effectively, providing real-time solutions that adapt to the variability inherent in renewable energy sources. Our reviews underscore the transformative potential of machine learning in making energy systems more efficient and sustainable.
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Pie chart analysis of the machine learning methods used in optimal power flow.</figcaption></figure>

The paper delves into the transformative role of ML in modernising OPF systems, highlighting a significant shift from traditional deterministic models to more dynamic, adaptive models equipped to handle the complexities introduced by the increasing use of renewable energy sources. This evolution fosters enhanced real-time decision-making and increases the resilience of power systems through the adoption of sophisticated ML techniques like DL and RL. These technologies not only adeptly manage the variability and unpredictability inherent in renewable sources such as wind and solar but also transform grid operations into intelligent, proactive management frameworks. This shift to a more anticipatory strategy improves the ability to forecast and react to changes in power flow, optimising both stability and efficiency. The integration of ML into OPF presents notable challenges, including high computational requirements and the critical dependence on the quality and availability of data. Furthermore, effective deployment of these advanced technologies requires supportive regulatory frameworks that facilitate innovation while ensuring alignment with broader objectives such as sustainability and public safety. As the paper indicates, embracing these challenges and opportunities is essential for developing power systems that are not only more efficient but also robust and adaptable to the evolving demands of energy management.

## 4. Conclusions

This paper reviews two vital aspects of renewable integration by exploring possibilities for advanced solutions from the generating device and power system operation perspectives. The review covers recent developments in thermal condition monitoring to examine how the capacity of existing renewable energy generators such as wind turbines can be expanded at low cost and in power flow optimisation with machine learning to examine how a low-carbon renewable dominated power network can be achieved. Utilisation of the thermal design margins for power equipment has the potential to expand renewable energy generation capacity. This is particularly true for wind power generation, where the uprating of existing wind turbines could increase the available renewable energy output. In situ fibre optic thermal sensing was identified as the technique that can facilitate the required thermal monitoring capability for WT generators; retrofitting
fibre optic sensors to in-service machinery in situ may be challenging in practice, and the application of effective thermal estimators, where available, would also be of interest in these cases. Furthermore, the development of dynamic and adaptive optimal power flow models can lead to a more informed and proactive management strategy for the power grids while optimising resource utilisation and minimising operational risks. With machine learning, the power grid can redefine its management boundaries and realise a platform for intelligent decision-making. System-level automation of this process by combining thermal condition monitoring with optimal power flow is highly desirable, yet it remains a challenge for control and management of renewable energy integration into the grid. The advanced sensor/sensing systems and machine learning approaches reviewed in this paper hold the potential to provide a viable and efficient solution to improve power capacity exploitation and maintain network stability in an economical and environmentally affordable way. However, considerable further research is needed to achieve this goal.

**Author Contributions:** Conceptualization, all authors; methodology, X.M., J.A. (Judith Apsley), A.G., J.A. (Jamal Aldahmashi), S.D. and M.B.; formal analysis A.G. and J.A. (Jamal Aldahmashi); investigation, A.G., J.A. (Jamal Aldahmashi); resources, S.D., J.A. (Judith Apsley) and X.M.; data curation, A.G. and J.A. (Jamal Aldahmashi); writing—original draft preparation, A.G. and J.A. (Jamal Aldahmashi); writing—review and editing J.A. (Judith Apsley), X.M., M.B. and S.D.; visualization, A.G. and J.A. (Jamal Aldahmashi); supervision, S.D., X.M., J.A. (Judith Apsley); project administration, S.D., J.A. (Judith Apsley) and X.M.; funding acquisition, J.A. (Judith Apsley), S.D., X.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** The authors are grateful to UKRI/EPSRC for providing a PhD stipend for Mr Aras Ghafoor as part of DTP scheme at the Department of Electrical and Electronic Engineering at University of Manchester. The authors are also grateful to the Deanship of Scientific Research at the Northern Border University, Arar, KSA, for funding a PhD studentship for Mr Jamal Aldahmashi through the project number "NBU-SAFIR-2024".

**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. RUSI. How Will Growth in Renewables Change the UK's Approach to Energy Security? Available online: https://rusi.org/explore-our-research/publications/commentary/how-will-growth-renewables-change-uks-approach-energy-security (accessed on 31 January 2024).

2. NEMA. NEMA MG-1 Motor-Generator Standard. Available online: https://law.resource.org/pub/us/cfr/ibr/005/nema.mg-1.2009.pdf (accessed on 12 June 2023).

3. Baker, N.; Liserre, M.; Dupont, L.; Avenas, Y. Improved reliability of power modules: A review of online junction temperature measurement methods. *IEEE Ind. Electron. Mag.* **2014**, *8*, 17–27. [CrossRef]

4. Siemens Gamesa. Asset Optimization Services. Available online: https://www.siemensgamesa.com/global/en/home/products-and-services/service-wind/asset-optimization.html (accessed on 8 April 2023).

5. Vestas. Fleet Optimisation. Available online: https://www.vestas.com/en/services/fleet-optimisation#accordion-d626793722-item-03a4e4001e (accessed on 5 April 2023).

6. Ghafoor, A.; Apsley, J.; Djurović, S. Thermal Margin Exploitation for Increased Energy Yield in Wind Turbine Permanent Magnet Synchronous Generators. In Proceedings of the 12th International Conference on Power Electronics, Machines and Drives (PEMD), Brussels, Belgium, 23–24 October 2023; pp. 1–6.

7. DNVGL. Certification of Condition Monitoring. Available online: https://www.dbassetservices.com/images/DNVGL-SE-0439.pdf (accessed on 12 June 2023).

8. Mohammed, A.; Djurović, S. Stator Winding Internal Thermal Monitoring and Analysis Using In Situ FBG Sensing Technology. *IEEE Trans. Energy Convers.* **2018**, *33*, 1508–1518. [CrossRef]

9. Suryandi, A.A.; Sarma, N.; Mohammed, A.; Peesapati, V.; Djurović, S. Fiber Optic Fiber Bragg Grating Sensing for Monitoring and Testing of Electric Machinery: Current State of the Art and Outlook. *Machines* **2022**, *10*, 1103. [CrossRef]

10. Liton Hossain, M.; Abu-Siada, A.; Muyeen, S.M. Methods for Advanced Wind Turbine Condition Monitoring and Early Diagnosis: A Literature Review. *Energies* **2018**, *11*, 1309. [CrossRef]

11. Fabian, M.; Hind, D.M.; Gerada, C.; Sun, T.; Grattan, K.T.V. Comprehensive Monitoring of Electrical Machine Parameters Using an Integrated Fiber Bragg Grating-Based Sensor System. *J. Light. Technol.* **2018**, *36*, 1046–1051. [CrossRef]

12. Chen, S.; Vilchis-Rodriguez, D.; Barnes, M.; Djurović, S. Direct On-Chip IGBT Thermal Sensing Using Adhesive Bonded FBG Sensors. *IEEE Sens. J.* **2023**, *23*, 22507–22516. [CrossRef]
13. Chen, S.; Vilchis-Rodriguez, D.; Djurovic, S.; Barnes, M.; McKeever, P.; Jia, C. Direct on Chip Thermal Measurement in IGBT Modules Using FBG Technology—Sensing Head Interfacing. *IEEE Sens. J.* **2022**, *22*, 1309–1320. [CrossRef]

14. Chen, S.; Vilchis-Rodriguez, D.; Barnes, M.; Djurovic, S. A Comparison of Chip Temperature Acquisition Technologies of IGBT Power Modules. *IEEE Sens. J.* **2024**, *24*, 19107–19116. [CrossRef]

15. Qi, F.; Ralev, I.; Stippich, A.; De Doncker, R.W. Model Predictive Overload Control of an Automotive Switched Reluctance Motor for Frequent Rapid Accelerations. In Proceedings of the 19th International Conference on Electrical Machines and Sytems (ICEMS), Chiba, Japan, 13–16 November 2016; pp. 1–6.

16. Sun, T.; Wang, J.; Griffo, A.; Sen, B. Active Thermal Management for Interior Permanent Magnet Synchronous Machine (IPMSM) Drives Based on Model Predictive Control. *IEEE Trans. Ind. Appl.* **2018**, *54*, 4506–4514. [CrossRef]

17. Qi, F.; Stippich, A.; Koschik, S.; De Doncker, R.W. Model Predictive Overload Control of Induction Motors. In Proceedings of the IEEE International Electric Machines and Drives Conference (IEMDC), Coeur d'Alene, ID, USA, 10–13 May 2015; pp. 999–1005.

18. Han, X.; Mu, C.; Yan, J.; Niu, Z. An Autonomous Control Technology Based on Deep Reinforcement Learning for Optimal Active Power Dispatch. *Int. J. Electr. Power Energy Syst.* **2023**, *14*, 108686. [CrossRef]

19. Woo, J.H.; Wu, L.; Park, J.B.; Roh, J.H. Real-Time Optimal Power Flow Using Twin Delayed Deep Deterministic Policy Gradient Algorithm. *IEEE Access* **2020**, *8*, 213611–213618. [CrossRef]

20. Cao, D.; Hu, W.; Xu, X.; Wu, Q.; Huang, Q.; Chen, Z.; Blaabjerg, F. Deep Reinforcement Learning Based Approach for Optimal Power Flow of Distribution Networks Embedded with Renewable Energy and Storage Devices. *J. Mod. Power Syst. Clean Energy* **2021**, *9*, 1101–1110. [CrossRef]

21. Al-Saffar, M.; Musilek, P. Reinforcement Learning-Based Distributed BESS Management for Mitigating Overvoltage Issues in Systems with High PV Penetration. *IEEE Trans. Smart Grid* **2020**, *11*, 2980–2994. [CrossRef]

22. Guo, P.; Infield, D.; Yang, X. Wind Turbine Generator Condition-Monitoring Using Temperature Trend Analysis. *IEEE Trans. Sustain. Energy* **2012**, *3*, 124–133. [CrossRef]

23. De Azevedo, H.D.M.; Araújo, A.M.; Bouchonneau, N. A Review of Wind Turbine Bearing Condition Monitoring: State of the Art and Challenges. *Renew. Sustain. Energy Rev.* **2016**, *56*, 368–379. [CrossRef]

24. Benbouzid, M.; Berghout, T.; Sarma, N.; Djurović, S.; Wu, Y.; Ma, X. Intelligent Condition Monitoring of Wind Power Systems: State of the Art Review. *Energies* **2021**, *14*, 5967. [CrossRef]

25. Yang, W.; Tavner, P.J.; Crabtree, C.J.; Feng, Y.; Qiu, Y. Wind Turbine Condition Monitoring: Technical & Commercial Challenges. *Wind Energy* **2014**, *17*, 673–693. [CrossRef]

26. Tavner, P.; Ran, L.; Crabtree, C. *Condition Monitoring of Rotating Electrical Machines*, 3rd ed.; Institution of Engineering and Technology: London, UK, 2020; ISBN 978-1-78561-866-6.

27. Siemens.Siplus CMS. Available online: https://cache.industry.siemens.com/dl/files/350/36815350/att_58847/v1/IONSIMATIC_S7_PN_OperatingInstructions_2011-09_EN.pdf (accessed on 5 July 2021).

28. Mohammed, A.; Djurovic, S. FBG Thermal Sensing Features for Hot Spot Monitoring in Random Wound Electric Machine Coils. *IEEE Sens. J.* **2017**, *17*, 3058–3067. [CrossRef]

29. Mohammed, A.; Djurovic, S. In-Situ Thermal and Mechanical Fibre Optic Sensing for in-Service Electric Machinery Bearing Condition Monitoring. In Proceedings of the IEEE International Conference on Electric Machines and Drives (IEMDC), San Diego, CA, USA, 12–15 May 2019; pp. 37–43.

30. Tchakoua, P.; Wamkeue, R.; Ouhrouche, M.; Slaoui-Hasnaoui, F.; Tameghe, T.A.; Ekemb, G. Wind Turbine Condition Monitoring: State-of-the-Art Review, New Trends, and Future Challenges. *Energies* **2014**, *7*, 2595–2630. [CrossRef]

31. Crabtree, C.; Zappalá, D.; Tavner, P. Survey of Commercially Available Condition Monitoring Systems for Wind Turbines. Available online: https://dro.dur.ac.uk/12497/1/12497.pdf?DDD10+ttsd23+dul4eg (accessed on 12 June 2023).

32. Mohammed, A.; Djurović, S. FBG Thermal Sensing Ring Scheme for Stator Winding Condition Monitoring in PMSMs. *Trans. Transp. Electrif.* **2019**, *5*, 1370–1382. [CrossRef]

33. Mohammed, A.; Djurović, S. Rotor Condition Monitoring Using Fibre Optic Sensing Technology. In Proceedings of the 10th IET International Conference on Power Electronics, Machines and Drives (PEMD), Nottingham, UK, 1–3 December 2020; pp. 1–6.

34. Mohammed, A.; Djurovic, S. Electric Machine Bearing Health Monitoring and Ball Fault Detection by Simultaneous Thermo-Mechanical Fibre Optic Sensing. *IEEE Trans. Energy Convers.* **2021**, *36*, 71–80. [CrossRef]

35. Shang, K.; Zhang, Y.; Galea, M.; Brusic, V.; Korposh, S. Fibre Optic Sensors for the Monitoring of Rotating Electric Machines: A Review. *Opt. Quantum Electron.* **2021**, *53*, 75. [CrossRef]

36. Madonna, V.; Giangrande, P.; Lusuardi, L.; Cavallini, A.; Gerada, C.; Galea, M. Thermal Overload and Insulation Aging of Short Duty Cycle, Aerospace Motors. *IEEE Trans. Ind. Electron.* **2020**, *67*, 2618–2629. [CrossRef]

37. IEEE 112-2017; IEEE Standard Test Procedure for Polyphase Induction Motors and Generators. IEEE: Piscataway, NJ, USA. Available online: https://ieeexplore.ieee.org/document/8291810 (accessed on 12 June 2023).

38. TOSHIBA MOTORS & DRIVES Temperature Rise. Available online: https://www.toshiba.com/tic/datafiles/app_notes/ Temperature_Rise_1605749858.pdf (accessed on 1 June 2023).

39. Werneck, M.M.; Célia, R.; Allil, B.; Ribeiro, B.A. Calibration and Operation of a Fibre Bragg Grating Temperature Sensing System in a Grid-Connected Hydrogenerator. *IET Sci. Meas. Technol.* **2013**, *7*, 59–68. [CrossRef]

40. Lemmens, J.; Vanassche, P.; Driesen, J. Optimal Control of Traction Motor Drives Under Electrothermal Constraints. *IEEE J. Emerg. Sel. Top. Power Electron.* **2014**, *2*, 249–263. [CrossRef]
41. Hales, K.; Spruce, C.; BJerregaard, S.L.; Rasmussen, P.O. Over-Rating Control of a Wind Turbine Generator. Available online: https://patentimages.storage.googleapis.com/65/03/8d/acc8695a2e154e/US10544779.pdf (accessed on 22 June 2024).

42. Ghafoor, A.; Djurović, S.; Apsley, J. A Coupled Electromagnetic-thermal Dynamic Model for Wind Turbine Permanent Magnet Synchronous Generator Operation Analysis. In Proceedings of the 32nd International Symposium Industrial Electronics (IEEE ISIE), Helsinki, Finland, 19–21 June 2023; pp. 1–6.

43. Dommel, H.W.; Tinney, W.F. Optimal Power Flow Solutions. *IEEE Trans. Power Appar. Syst.* **1968**, *10*, 1866–1876. [CrossRef]

44. Ghaddar, B.; Marecek, J.; Mevissen, M. Optimal Power Flow as a Polynomial Optimization Problem. *IEEE Trans. Power Syst.* **2015**, *31*, 539–546. [CrossRef]

45. Grover-Silva, E.; Heleno, M.; Mashayekh, S.; Cardoso, G.; Girard, R.; Kariniotakis, G. A Stochastic Optimal Power Flow for Scheduling Flexible Resources in Microgrids Operation. *Appl. Energy* **2018**, *229*, 201–208. [CrossRef]

46. Madrigal, M.; Ponnambalam, K.; Quintana, V.H. Probabilistic Optimal Power Flow. In Proceedings of the IEEE Canadian Conference on Electrical and Computer Engineering, Waterloo, ON, Canada, 25–28 May 1998.

47. Reddy, S.S.; Bijwe, P.R.; Abhyankar, A.R. Faster Evolutionary Algorithm Based Optimal Power Flow Using Incremental Variables. *Int. J. Electr. Power Energy Syst.* **2014**, *54*, 198–210. [CrossRef]

48. Kang, Q.; Zhou, M.; An, J.; Wu, Q. Swarm Intelligence Approaches to Optimal Power Flow Problem with Distributed Generator Failures in Power Networks. *IEEE Trans. Autom. Sci. Eng.* **2012**, *10*, 343–353. [CrossRef]

49. Sutton, R.S.; Barto, A.G. *Reinforcement Learning: An Introduction*; MIT Press: Cambridge, MA, USA, 2018.

50. Bertsimas, D.; Litvinov, E.; Sun, X.A.; Zhao, J.; Zheng, T. Adaptive Robust Optimization for the Security Constrained Unit Commitment Problem. *IEEE Trans. Power Syst.* **2012**, *28*, 52–63. [CrossRef]

51. Jordan, M.I.; Mitchell, T.M. Machine Learning: Trends, Perspectives, and Prospects. *Science* **2015**, *349*, 255–260. [CrossRef] [PubMed]

52. Mousavi, S.S.; Schukat, M.; Howley, E. Deep Reinforcement Learning: An Overview. In *Proceedings of the SAI Intelligent Systems Conference (IntelliSys) 2016*; Springer International Publishing; Berlin/Heidelberg, Germany, 2018.

53. Pham, Q.V.; Liyanage, M.; Deepa, N.; VVSS, M.; Reddy, S.; Maddikunta, P.K.R.; Hwang, W.J. Deep Learning for Intelligent Demand Response and Smart Grids: A Comprehensive Survey. *arXiv* **2021**, preprint. [CrossRef]

54. Aldahmashi, J.; Ma, X. Real-Time Energy Management in Smart Homes Through Deep Reinforcement Learning. *IEEE Access* **2024**, *12*, 43155–43172. [CrossRef]

55. Ullah, Z.; Wang, S.; Wu, G.; Hasanien, H.M.; Jabbar, M.W.; Qazi, H.S.; Elkadeem, M.R. Advanced Studies for Probabilistic Optimal Power Flow in Active Distribution Networks: A Scientometric Review. *IET Gener. Transm. Distrib.* **2022**, *16*, 3579–3604. [CrossRef]

56. Yan, Z.; Xu, Y. Real-Time Optimal Power Flow: A Lagrangian Based Deep Reinforcement Learning Approach. *IEEE Trans. Power Syst.* **2020**, *35*, 3270–3273. [CrossRef]

57. da Silva André, J.; Stai, E.; Stanojev, O.; Hug, G. Battery Control with Lookahead Constraints in Distribution Grids Using Reinforcement Learning. *Electr. Power Syst. Res.* **2022**, *211*, 108551. [CrossRef]

58. Alabdullah, M.H.; Abido, M.A. Microgrid Energy Management Using Deep Q-Network Reinforcement Learning. *Alex. Eng. J.* **2022**, *61*, 9069–9078. [CrossRef]

59. Hua, H.; Qin, Z.; Dong, N.; Qin, Y.; Ye, M.; Wang, Z.; Cao, J. Data-Driven Dynamical Control for Bottom-Up Energy Internet System. *IEEE Trans. Sustain. Energy* **2021**, *13*, 315–327. [CrossRef]

60. Zhang, B.; Hu, W.; Xu, X.; Li, T.; Zhang, Z.; Chen, Z. Physical-Model-Free Intelligent Energy Management for a Grid-Connected Hybrid Wind-Microturbine-PV-EV Energy System Via Deep Reinforcement Learning Approach. *Renew. Energy* **2022**, *200*, 433–448. [CrossRef]

61. Jin, J.; Xu, Y. Optimal Policy Characterization Enhanced Actor-Critic Approach for Electric Vehicle Charging Scheduling in a Power Distribution Network. *IEEE Trans. Smart Grid* **2020**, *12*, 1416–1428. [CrossRef]

62. Xing, Q.; Chen, Z.; Zhang, T.; Li, X.; Sun, K. Real-Time Optimal Scheduling for Active Distribution Networks: A Graph Reinforcement Learning Method. *Int. J. Electr. Power Energy Syst.* **2023**, *145*, 108637. [CrossRef]

63. Wang, B.; Li, Y.; Ming, W.; Wang, S. Deep Reinforcement Learning Method for Demand Response Management of Interruptible Load. *IEEE Trans. Smart Grid* **2020**, *11*, 3146–3155. [CrossRef]

64. Bui, V.H.; Su, W. Real-Time Operation of Distribution Network: A Deep Reinforcement Learning-Based Reconfiguration Approach. *Sustain. Energy Technol. Assess.* **2022**, *50*, 101841. [CrossRef]

65. Gao, Y.; Wang, W.; Shi, J.; Yu, N. Batch-Constrained Reinforcement Learning for Dynamic Distribution Network Reconfiguration. *IEEE Trans. Smart Grid* **2020**, *11*, 5357–5369. [CrossRef]

66. Aldahmashi, J.; Ma, X. Advanced Machine Learning Approach of Power Flow Optimization in Community Microgrid. In Proceedings of the 2022 27th International Conference on Automation and Computing (ICAC), Bristol, UK, 1–3 September 2022; pp. 1–6.

67. Li, X.; Han, X.; Yang, M. Day-Ahead Optimal Dispatch Strategy for Active Distribution Network Based on Improved Deep Reinforcement Learning. *IEEE Access* **2022**, *10*, 9357–9370. [CrossRef]

68. Kushwaha, D.S.; Biron, Z.; Abdollahi, R. Economic Energy Dispatch of Microgrid Using DeepLSTM-Based Deep Reinforcement Learning. In Proceedings of the 2022 IEEE Power & Energy Society General Meeting (PESGM), Denver, CO, USA, 17–21 July 2022; pp. 1–5.
69. Li, X.; Wang, J.; Lu, Z.; Cai, Y. A Cloud Edge Computing Method for Economic Dispatch of Active Distribution Network with Multi-Microgrids. *Electr. Power Syst. Res.* **2023**, *214*, 108806. [CrossRef]

70. Gao, Z.; Zheng, Z.; Wu, J.; Qi, L.; Li, W.; Yang, Y. Reactive Power Optimization of Distribution Network Based on Deep Reinforcement Learning and Multi Agent System. In Proceedings of the 2021 IEEE 5th Conference on Energy Internet and Energy System Integration (EI2), Taiyuan, China, 22–24 October 2021; pp. 1052–1057.

71. Yang, Z.; Yang, F.; Min, H.; Tian, H.; Hu, W.; Liu, J.; Eghbalian, N. Energy Management Programming to Reduce Distribution Network Operating Costs in the Presence of Electric Vehicles and Renewable Energy Sources. *Energy* **2023**, *263*, 125695. [CrossRef]

72. Hai, D.; Zhu, T.; Duan, S.; Huang, W.; Li, W. Deep Reinforcement Learning for Volt/VAR Control in Distribution Systems: A Review. In Proceedings of the 2022 5th International Conference on Energy, Electrical and Power Engineering (CEEPE) IEEE, Chongqing, China, 22–24 April 2022; pp. 596–601.

73. Sun, X.; Qiu, J.; Zhao, J. Optimal Local Volt/VAR Control for Photovoltaic Inverters in Active Distribution Networks. *IEEE Trans. Power Syst.* **2021**, *36*, 5756–5766. [CrossRef]

74. Beyer, K.; Beckmann, R.; Geißendörfer, S.; von Maydell, K.; Agert, C. Adaptive Online-Learning Volt-VAR Control for Smart Inverters Using Deep Reinforcement Learning. *Energies* **2021**, *14*, 1991. [CrossRef]

75. Cao, D.; Hu, W.; Zhao, J.; Huang, Q.; Chen, Z.; Blaabjerg, F. A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters. *IEEE Trans. Power Syst.* **2020**, *35*, 4120–4123. [CrossRef]

76. Liu, H.; Wu, W. Two-Stage Deep Reinforcement Learning for Inverter-Based Volt-VAR Control in Active Distribution Networks. *IEEE Trans. Smart Grid* **2020**, *12*, 2037–2047. [CrossRef]

77. Zhou, Y.; Zhang, B.; Xu, C.; Lan, T.; Diao, R.; Shi, D.; Wang, Z.; Lee, W.-J. A Data-Driven Method for Fast AC Optimal Power Flow Solutions Via Deep Reinforcement Learning. *J. Mod. Power Syst. Clean Energy* **2020**, *8*, 1128–1139. [CrossRef]

78. Tan, K.M.; Babu, T.S.; Ramachandaramurthy, V.K.; Kasinathan, P.; Solanki, S.G.; Raveendran, S.K. Empowering Smart Grid: A Comprehensive Review of Energy Storage Technology and Application with Renewable Energy Integration. *J. Energy Storage* **2021**, *39*, 102591. [CrossRef]

79. Zhou, W.; Zhang, N.; Cao, Z.; Chen, Y.; Wang, M.; Liu, Y. Voltage Regulation Based on Deep Reinforcement Learning Algorithm in Distribution Network with Energy Storage System. In Proceedings of the 2021 4th International Conference on Energy, Electrical and Power Engineering (CEEPE) IEEE, Chongqing, China, 23–25 April 2021; pp. 892–896.

80. Wang, Y.; Mao, M.; Chang, L.; Hatziargyriou, N.D. Intelligent Voltage Control Method in Active Distribution Networks Based on Averaged Weighted Double Deep Q-Network Algorithm. *J. Mod. Power Syst. Clean Energy* **2022**, *11*, 132–143. [CrossRef]

81. Kelker, M.; Quakernack, L.; Haubrock, J. Multi agent deep Q-reinforcement learning for autonomous low voltage grid control. In In Proceedings of the 2021 IEEE PES Innovative Smart Grid Technologies Europe (ISGT Europe), Espoo, Finland, 18–21 October 2021; pp. 1–6.

82. Hossain, R.; Lakouraj, M.M.; Ghasemkhani, A.; Livani, H.; Ben-Idris, M. Deep Reinforcement Learning-Based Volt-VAR Optimization in Distribution Grids with Inverter-Based Resources. In Proceedings of the 2021 North American Power Symposium (NAPS) IEEE, College Station, TX, USA, 14–16 November 2021; pp. 1–6.

83. Moy, K.; Tae, C.; Wang, Y.; Henri, G.; Bambos, N.; Rajagopal, R. An OpenAI-OpenDSS Framework for Reinforcement Learning on Distribution-Level Microgrids. In Proceedings of the 2021 IEEE Power & Energy Society General Meeting (PESGM) IEEE, Washington, DC, USA, 25–29 July 2021; pp. 1–5.

84. Yang, Q.; Wang, G.; Sadeghi, A.; Giannakis, G.B.; Sun, J. Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning. *IEEE Trans. Smart Grid* **2019**, *11*, 2313–2323. [CrossRef]

85. Tahir, Y.; Khan, M.F.N.; Sajjad, I.A.; Martirano, L. Optimal Control of Active Distribution Network Using Deep Reinforcement Learning. In Proceedings of the 2022 IEEE International Conference on Environment and Electrical Engineering and 2022 IEEE Industrial and Commercial Power Systems Europe (EEEIC/I&CPS Europe), Prague, Czech Republic, 28 June–1 July 2022; pp. 1–6.

86. Wang, W.; Yu, N.; Gao, Y.; Shi, J. Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems. *IEEE Trans. Smart Grid* **2019**, *11*, 3008–3018. [CrossRef]

87. Li, W.; Huang, W.; Zhu, T.; Wu, M.; Yan, Z. Deep Reinforcement Learning Based Continuous Volt-VAR Optimization in Power Distribution Systems with Renewable Energy Resources. In Proceedings of the 2021 IEEE Sustainable Power and Energy Conference (iSPEC), Nanjing, China, 23–25 December 2021; pp. 682–686.

88. Cao, D.; Zhao, J.; Hu, W.; Ding, F.; Huang, Q.; Chen, Z.; Blaabjerg, F. Data-Driven Multi-Agent Deep Reinforcement Learning for Distribution System Decentralized Voltage Control with High Penetration of PVs. *IEEE Trans. Smart Grid* **2021**, *12*, 4137–4150. [CrossRef]

89. Kundačina, O.B.; Vidović, P.M.; Petković, M.R. Solving Dynamic Distribution Network Reconfiguration Using Deep Reinforcement Learning. *Electr. Eng.* **2022**, *104*, 1487–1501. [CrossRef]

90. Rahman, J.; Jacob, R.A.; Paul, S.; Chowdhury, S.; Zhang, J. Reinforcement Learning Enabled Microgrid Network Reconfiguration Under Disruptive Events. In Proceedings of the 2022 IEEE Kansas Power and Energy Conference (KPEC), Manhattan, KS, USA, 25–26 April 2022; pp. 1–6.

91. Zhang, J.; Lu, C.; Fang, C.; Ling, X.; Zhang, Y. Load Shedding Scheme with Deep Reinforcement Learning to Improve Short-Term Voltage Stability. In Proceedings of the 2018 IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia), Beijing, China, 22–25 May 2018; pp. 13–18.
92. Zhang, B.; Hu, W.; Ghias, A.M.Y.M.; Xu, X.; Chen, Z. Multi-Agent Deep Reinforcement Learning-Based Coordination Control for Grid-Aware Multi-Buildings. *Appl. Energy* **2022**, *328*, 120215. [CrossRef]

93. Xu, Y.; Zhang, W.; Liu, W.; Ferrese, F. Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch. *IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.)* **2012**, *42*, 1742–1751. [CrossRef]

94. Kou, P.; Liang, D.; Wang, C.; Wu, Z.; Gao, L. Safe Deep Reinforcement Learning-Based Constrained Optimal Control Scheme for Active Distribution Networks. *Appl. Energy* **2020**, *264*, 114772. [CrossRef]

95. Diao, R.; Wang, Z.; Shi, D.; Chang, Q.; Duan, J.; Zhang, X. Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning. In Proceedings of the 2019 IEEE Power & Energy Society General Meeting (PESGM), Atlanta, GA, USA, 4–8 August 2019; pp. 1–5.

96. Cao, D.; Zhao, J.; Hu, W.; Yu, N.; Ding, F.; Huang, Q.; Chen, Z. Deep Reinforcement Learning Enabled Physical-Model-Free Two-Timescale Voltage Control Method for Active Distribution Systems. *IEEE Trans. Smart Grid* **2021**, *13*, 149–165. [CrossRef]

97. Sun, X.; Qiu, J. A Customized Voltage Control Strategy for Electric Vehicles in Distribution Networks with Reinforcement Learning Method. *IEEE Trans. Ind. Inform.* **2021**, *17*, 6852–6863. [CrossRef]

98. Zhang, J.; Lu, C.; Si, J.; Song, J.; Su, Y. Deep Reinforcement Learning for Short-Term Voltage Control by Dynamic Load Shedding in China Southern Power Grid. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 8–13 July 2018; pp. 1–8.

99. Kocer, M.C.; Cengiz, C.; Gezer, M.; Gunes, D.; Cinar, M.A.; Alboyaci, B.; Onen, A. Assessment of Battery Storage Technologies for a Turkish Power Network. *Sustainability* **2019**, *11*, 3669. [CrossRef]

100. Wei, T.; Chu, X.; Yang, D.; Ma, H. Power Balance Control of RES Integrated Power System by Deep Reinforcement Learning with Optimized Utilization Rate of Renewable Energy. *Energy Rep.* **2022**, *8*, 544–553. [CrossRef]

101. Kabir, F.; Gao, Y.; Yu, N. Reinforcement Learning-Based Smart Inverter Control with Polar Action Space in Power Distribution Systems. In Proceedings of the 2021 IEEE Conference on Control Technology and Applications (CCTA), San Diego, CA, USA, 8–11 August 2021; pp. 315–322.

102. Pei, Y.; Yao, Y.; Zhao, J.; Ding, F.; Wang, J. Multi-Agent Deep Reinforcement Learning for Realistic Distribution System Voltage Control Using PV Inverters. In Proceedings of the 2022 IEEE Power & Energy Society General Meeting (PESGM), Denver, CO, USA, 17–21 July 2022; pp. 1–5.

103. Zhu, Z.; Weng, Z.; Zheng, H. Optimal Operation of a Microgrid with Hydrogen Storage Based on Deep Reinforcement Learning. *Electronics* **2022**, *11*, 196. [CrossRef]

104. Qiu, D.; Wang, Y.; Sun, M.; Strbac, G. Multi-Service Provision for Electric Vehicles in Power-Transportation Networks Towards a Low-Carbon Transition: A Hierarchical and Hybrid Multi-Agent Reinforcement Learning Approach. *Appl. Energy* **2022**, *313*, 118790. [CrossRef]

105. Gao, Y.; Shi, J.; Wang, W.; Yu, N. Dynamic Distribution Network Reconfiguration Using Reinforcement Learning. In Proceedings of the 2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), Beijing, China, 21–23 October 2019; pp. 1–7.

106. Li, Y.; Hao, G.; Liu, Y.; Yu, Y.; Ni, Z.; Zhao, Y. Many-Objective Distribution Network Reconfiguration via Deep Reinforcement Learning Assisted Optimization Algorithm. *IEEE Trans. Power Deliv.* **2021**, *37*, 2230–2244. [CrossRef]

107. Zhang, X.; Liu, Y.; Duan, J.; Qiu, G.; Liu, T.; Liu, J. DDPG-Based Multi-Agent Framework for SVC Tuning in Urban Power Grid with Renewable Energy Resources. *IEEE Trans. Power Syst.* **2021**, *36*, 5465–5475. [CrossRef]

108. Zhou, Y.; Lee, W.-J.; Diao, R.; Shi, D. Deep Reinforcement Learning Based Real-Time AC Optimal Power Flow Considering Uncertainties. *J. Mod. Power Syst. Clean Energy* **2021**, *10*, 1098–1109. [CrossRef]

109. Jung, Y.; Han, C.; Lee, D.; Song, S.; Jang, G. Adaptive Volt-Var Control in Smart PV Inverter for Mitigating Voltage Unbalance at PCC Using Multiagent Deep Reinforcement Learning. *Appl. Sci.* **2021**, *11*, 8979. [CrossRef]

110. Zhu, J.; Hu, W.; Xu, X.; Liu, H.; Pan, L.; Fan, H.; Zhang, Z.; Chen, Z. Optimal Scheduling of a Wind Energy Dominated Distribution Network Via a Deep Reinforcement Learning Approach. *Renew. Energy* **2022**, *201*, 792–801. [CrossRef]

111. Zhang, B.; Hu, W.; Li, J.; Cao, D.; Huang, R.; Huang, Q.; Chen, Z.; Blaabjerg, F. Dynamic Energy Conversion and Management Strategy for an Integrated Electricity and Natural Gas System with Renewable Energy: Deep Reinforcement Learning Approach. *Energy Convers. Manag.* **2020**, *220*, 113063. [CrossRef]

112. Velamuri, S.; Kantipudi, M.V.V.P.; Sitharthan, R.; Kanakadhurga, D.; Prabaharan, N.; Rajkumar, A. A Q-Learning Based Electric Vehicle Scheduling Technique in a Distribution System for Power Loss Curtailment. *Sustain. Comput. Inform. Syst.* **2022**, *36*, 100798. [CrossRef]

113. Wang, S.; Du, L.; Fan, X.; Huang, Q. Deep Reinforcement Scheduling of Energy Storage Systems for Real-Time Voltage Regulation in Unbalanced LV Networks with High PV Penetration. *IEEE Trans. Sustain. Energy* **2021**, *12*, 2342–2352. [CrossRef]

114. Huang, R.; Chen, Y.; Yin, T.; Li, X.; Li, A.; Tan, J.; Yu, W.; Liu, Y.; Huang, Q. Accelerated Derivative-Free Deep Reinforcement Learning for Large-Scale Grid Emergency Voltage Control. *IEEE Trans. Power Syst.* **2021**, *37*, 14–25. [CrossRef]

115. Li, J.; Chen, S.; Wang, X.; Pu, T. Load Shedding Control Strategy in Power Grid Emergency State Based on Deep Reinforcement Learning. *CSEE J. Power Energy Syst.* **2021**, *8*, 1175–1182. [CrossRef]

116. Matavalam, A.R.R.; Guddanti, K.P.; Weng, Y.; Ajjarapu, V. Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading. *IEEE Trans. Power Syst.* **2022**, *38*, 4206–4220. [CrossRef]
117. Oh, S.H.; Yoon, Y.T.; Kim, S.W. Online Reconfiguration Scheme of Self-Sufficient Distribution Network Based on a Reinforcement Learning Approach. *Appl. Energy* **2020**, *280*, 115900. [CrossRef]

118. Nguyen, H.T.; Choi, D.-H. Three-Stage Inverter-Based Peak Shaving and Volt-VAR Control in Active Distribution Networks Using Online Safe Deep Reinforcement Learning. *IEEE Trans. Smart Grid* **2022**, *13*, 3266–3277. [CrossRef]

119. Horri, R.; Roudsari, H.M. Reinforcement-Learning-Based Load Shedding and Intentional Voltage Manipulation Approach in a Microgrid Considering Load Dynamics. *IET Gener. Transm. Distrib.* **2022**, *16*, 3384–3401. [CrossRef]

120. Hagmar, H.; Tuan, L.A.; Eriksson, R. Deep Reinforcement Learning for Long-Term Voltage Stability Control. *arXiv* **2022**, Preprint. [CrossRef]

121. Ma, Q.; Zhang, H.; He, X.; Tang, J.; Yuan, X.; Wang, G. Emergency Frequency Control Strategy Using Demand Response Based on Deep Reinforcement Learning. In Proceedings of the 2020 12th IEEE PES Asia-Pacific Power and Energy Engineering Conference (APPEEC), Nanjin, China, 20–23 September 2020; pp. 1–5.

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.