# Optimization Method of Power Equipment Maintenance Plan Decision-Making Based on Deep Reinforcement Learning

Authors: Yanhua Yang; Ligang Yao
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

The safe and reliable operation of power grid equipment is the basis for ensuring the safe operation of the power system. At present, the traditional periodical maintenance has exposed the abuses such as deficient maintenance and excess maintenance. Based on a multiagent deep reinforcement learning decision-making optimization algorithm, a method for decision-making and optimization of power grid equipment maintenance plans is proposed. In this paper, an optimization model of power grid equipment maintenance plan that takes into account the reliability and economics of power grid operation is constructed with maintenance constraints and power grid safety constraints as its constraints. The deep distributed recurrentQ-networks multiagent deep reinforcement learning is adopted to solve the optimization model. The deep distributed recurrent Q-networks multiagent deep reinforcement learning uses the high-dimensional feature extraction capabilities of deep learning and decision-making capabilities of reinforcement learning to solve the multiobjective decision-making problem of power grid maintenance planning. Through case analysis, the comparative results show that the proposed algorithm has better optimization and decision-making ability, as well as lower maintenance cost. Accordingly, the algorithm can realize the optimal decision of power grid equipment maintenance plan. The expected value of power shortage and maintenance cost obtained by the proposed method is $71.75$ $MW·H$ and $496000$ $yuan$.


---

## Body

## 1. Introduction

In order to ensure the continuous and safe operation of power equipment, it is necessary to do a good job in preventive plan maintenance and repair of equipment and timely find and eliminate potential accidents. The arrangement of power equipment maintenance plan is an important measure to ensure the safe operation of power grid, reliable power supply, and improve the health level of equipment. With the continuous development of the power system, equipment maintenance is facing new challenges [1]. How to optimize maintenance plan, prevent over maintenance of equipment, save the maintenance cost, and improve the reliability of planned maintenance power supply is problem which is still have to be solved. On the one hand, the increasing number of power equipment makes the maintenance workload heavier, further increasing the maintenance cost. How to optimize the maintenance plan, prevent over maintenance of equipment, and reduce the maintenance cost is one of the goals to achieve the economic operation of the power grid. On the other hand, the economic development and people's daily life put forward higher requirements for power supply reliability, and planned maintenance is one of the reasons for the reduction of power supply reliability. Therefore, the optimization of equipment maintenance plan and reasonable arrangement of equipment maintenance can ensure the safe and reliable operation of power equipment and improve the reliability of power supply.The maintenance plan of power equipment depends on manual preparation, which is limited by the professional technical ability and working experience of the personnel. According to the requirements of the power grid operation mode, dispatching plan, and equipment maintenance regulations, maintenance plan preparation personnel prepares maintenance plan to meet the safe and reliable operation of power grid. However, with the development of the power grid, some problems are gradually exposed in the manual maintenance plan; reliability and economy cannot be guaranteed; many factors need to be considered in the maintenance plan formulation, such as heavy workload and low efficiency; manual planning is not easy to analyze data statistics; the quality of maintenance plan is affected by the technical ability and work experience of professional personnel.Maintenance plan optimization is a multistage dynamic planning process [2]. The solving algorithm for optimization models includes mathematical optimization method and intelligent optimization method. There are two kinds of optimization models: single objective function model and multiobjective function model. In the economic single objective function model, for example, literature [3] maximizes demand reliability by minimizing total square sum of reserves (SSR) and power production cost (mainly fuel cost) and uses the multiobjective simulated annealing method based on dominance to determine the compromise solution of the model. In [4], the optimization model is established based on the index of network loss, aiming at the minimum of monthly network loss. In the reliability single objective function model, for example, literature [5] puts forward an optimization model of distribution equipment maintenance based on risk assessment, which takes the minimum risk of power grid operation as the optimization objective and uses the particle swarm optimization algorithm to solve it. Based on the risk assessment, literature [6] puts forward the coordinated maintenance strategy for UHV receiving power equipment, which provides reference for ensuring the normal operation of UHV. In [7], based on the equipment status and the risk of power grid loss, the optimization model of power distribution equipment maintenance plan is established. In [8], a maintenance rate optimization model of power equipment asset management is proposed, which uses Markov process and explicitly considers the aging of equipment. In the literature, the algorithm used to solve the problem stays in single objective optimization but does not achieve multiobjective optimization; the proposed optimization algorithm only optimizes the single objective and does not coordinate the economy and reliability of the maintenance plan.In [9], an improved multiobjective differential evolution algorithm for adaptive optimization of control parameters to solve high-speed rail line planning is proposed, as well as a heuristic algorithm to obtain a better initial solution. In [10], according to application of multiagent reinforcement learning to guide multi workflow scheduling on service cloud, an optimization algorithm is proposed, which takes the number of workflow applications and heterogeneous virtual machines as state input and takes the maximum completion time and cost as rewards. In [11], based on the radial basis function neural network approximation model, a fast multiobjective optimization algorithm for contactor characteristics is proposed, which can speed up the convergence and get optimal solution. In [12], a multiobjective optimization method based on sensitivity is proposed to optimal configuration of thyristor-controlled series capacitors in the transmission network, which can reduce the total reactive power loss and reduce the load capacity of the transmission line. In reference [13], a coordinated stochastic scheduling model based on the multiobjective optimization method is proposed to improve the absorption capacity of wind power on the premise of energy saving and emission reduction. The application of deep learning and reinforcement learning in multiobjective optimization provides a new idea for decision-making optimization of power equipment maintenance plan.The development and application of AI [9], big data [10], parallel computing, and optimization algorithm and theory [14, 15] create favorable conditions for the optimization of power equipment maintenance plan. Deep learning [16] (DL) is composed of multilayer nonlinear units, which can automatically learn abstract features from training data; reinforcement learning (RL) [17] has strong decision-making ability, high adaptability, and advantages of decision optimization; deep reinforcement learning (DRL) [18] uses the high-dimensional feature extraction ability of DL and the decision-making ability of RL to solve the decision-making problems in high-dimensional state space and high-dimensional action space [19–21]. The application of the deep reinforcement learning method in the power system is mainly in intelligent power generation control, power grid intelligent control, and other fields [1, 22, 23].Deep reinforcement learning is an end-to-end perception and decision-making system, which has universality. The process of perceptual learning and decision-making optimization is as follows: the agent interacts with the environment to get a high-dimensional observation states, and uses the deep learning method to get specific feature representation; based on the reward evaluation value function, the current state is mapped to the corresponding action by decision; the environment reacts to the action and gets the next observation. Finally, the optimal decision of the target can be obtained by continuously cycling the above process.In this paper, a multiobjective mathematical model of power equipment maintenance optimization considering reliability and economy is established, and a kind of deep distributed Q-network is adopted (Q-networks, DDRQN) to realize multiagent deep reinforcement learning solution, to make full use of DRL’s optimization ability and decision-making ability, and to realize the intelligent arrangement of power equipment maintenance plan. The comparison experiment has processed the new method and single agent deep reinforcement learning algorithm as well as the particle swarm intelligence algorithm for the deep cyclicQ-network. The results show that the proposed algorithm has higher reliability, lower maintenance cost, and more reasonable maintenance plan.


## 2. Optimization Model of Power Equipment Maintenance Plan

### 2.1. Objective Function

Power equipment maintenance plan optimization is a multiobjective and multiconstraint optimization problem. The objective function can be divided into three categories: reliability objective function, economy objective function, and practicability objective function [24]. The reliability objective functions include loss of load probability (LOLP), minimum load loss due to power failure, maximum system reliability index, and minimum expected energy not supplied (Expected Energy Not Supplied, EENS). The economic objective function includes maintenance cost and outage loss. The practical goal is to put forward due maintenance, reduce maintenance outage, and even distribution of maintenance workload from the practical point of view.EENS is defined as the sum of the power loss caused by the outage of the power equipment. It evaluates the power supply reliability reduction caused by the outage of the equipment. The objective function expression is as follows:(1)f1=∑t=1T∑x∈StCx∏i=1MPixi1−Pi1−xiTt,where t refers to the number of maintenance periods, St refers to a collection of states, x=x1,x2,…,xn means the state vector of the equipment, Cx refers to the load shedding under the fault state, M refers to the number of equipment, xi=1 refers to the shutdown state, xi=0 means the equipment is in the running state, Pi refers to the equipment outage probability, Tt refers to the number of unit hours, and the unit is MW·H.The maintenance cost of the equipment refers to the expenses incurred by the maintenance of the equipment during the maintenance period. The objective function expression is as follows:(2)f2=∑i=1N∑t=1Tpitzituit,where N is the total number of equipment to be overhauled; pit represents the unit time cost; zit represents the maintenance team arranged for the equipment i in the t period; uit=1 represents the outage maintenance of the equipment i in the t period, uit=0 represents the normal operation, and unit is 10000 yuan.


### 2.2. Constraints

#### 2.2.1. Time Constraints

Any maintenance work shall be completed on time.(3)uit1si≤t≤si+mi,0,where uit refers to the equipment i maintenance status, uit=1 means the equipment i is powered off for maintenance in the t period, uit=0 means the equipment i is in normal operation for maintenance in the t period, si means the equipment i starts maintenance in the si period, mi is the maintenance period, ei means the earliest time to start maintenance, and li means the latest time to start maintenance.


#### 2.2.2. Maintenance Resource Constraints

In a maintenance cycle, the number of equipment that can be maintained at the same time is limited.(4)∑i=1Nzituit≤Zmax,where Zmax indicates the upper limit of maintenance workload in the t period.


#### 2.2.3. Simultaneous Maintenance Constraints

Equipment maintenance shall avoid repeated power failure at the same load point, and all problems shall be solved during power failure maintenance.(5)sk=si,where sk is the time for equipment k to start maintenance and si is the time for equipment i to start maintenance.


#### 2.2.4. Mutually Exclusive Maintenance Constraints

To prevent the expansion of power outage, some power equipment cannot be arranged for maintenance at the same time.(6)sj>si+mi+1,where sj is the time when equipment J starts maintenance; si is the time when equipment i starts maintenance; mi is the maintenance period of equipment i.


#### 2.2.5. Security Constraints

In order to ensure the safe and stable operation of power grid, the maintenance plan must be checked by power flow calculation.(7)Ii≤Ii,max,(8)Vi,min≤Vi≤Vi,max,(9)Pi≤Pi,max,where Ii,max, Vi,max, Vi,min, and Pi,max are, respectively, expressed as the upper and lower limit of current flow, the upper and lower limit of node voltage, and the allowable power limit. Ii is the current flowing through equipment I, Vi is the node voltage amplitude, and Pi is the active power passing through the device.


### 2.3. Optimization Model

In this paper, the expected value of energy shortage (EENS) is taken as the reliability evaluation index, and the maintenance cost is taken as the economic evaluation index of maintenance plan.(10)minfiX,s.t.giX=0,hjX≤0,where fiX represents the number i objective function of power equipment maintenance plan, X is the n-dimensional decision vector, m means number of objective function, giX and hjX, respectively, represents the equality constraint function and inequality constraint function, p is the number of equality constrained functions, and q is the number of inequality constrained function.In most cases, the economic objective function and reliability objective function are in conflict with each other. The improvement of one objective may lead to the performance degradation of the other, and it is impossible to make both objectives reach the optimal solution at the same time. It can only be dealt with in the middle of them to make each subobjective reach the optimization as much as possible. The solution of a multiobjective optimization problem is usually a set of solutions that meet the conditions. For the solution set, we can find a set of solutions that are as close to the best as possible.


## 2.1. Objective Function

Power equipment maintenance plan optimization is a multiobjective and multiconstraint optimization problem. The objective function can be divided into three categories: reliability objective function, economy objective function, and practicability objective function [24]. The reliability objective functions include loss of load probability (LOLP), minimum load loss due to power failure, maximum system reliability index, and minimum expected energy not supplied (Expected Energy Not Supplied, EENS). The economic objective function includes maintenance cost and outage loss. The practical goal is to put forward due maintenance, reduce maintenance outage, and even distribution of maintenance workload from the practical point of view.EENS is defined as the sum of the power loss caused by the outage of the power equipment. It evaluates the power supply reliability reduction caused by the outage of the equipment. The objective function expression is as follows:(1)f1=∑t=1T∑x∈StCx∏i=1MPixi1−Pi1−xiTt,where t refers to the number of maintenance periods, St refers to a collection of states, x=x1,x2,…,xn means the state vector of the equipment, Cx refers to the load shedding under the fault state, M refers to the number of equipment, xi=1 refers to the shutdown state, xi=0 means the equipment is in the running state, Pi refers to the equipment outage probability, Tt refers to the number of unit hours, and the unit is MW·H.The maintenance cost of the equipment refers to the expenses incurred by the maintenance of the equipment during the maintenance period. The objective function expression is as follows:(2)f2=∑i=1N∑t=1Tpitzituit,where N is the total number of equipment to be overhauled; pit represents the unit time cost; zit represents the maintenance team arranged for the equipment i in the t period; uit=1 represents the outage maintenance of the equipment i in the t period, uit=0 represents the normal operation, and unit is 10000 yuan.


## 2.2. Constraints

### 2.2.1. Time Constraints

Any maintenance work shall be completed on time.(3)uit1si≤t≤si+mi,0,where uit refers to the equipment i maintenance status, uit=1 means the equipment i is powered off for maintenance in the t period, uit=0 means the equipment i is in normal operation for maintenance in the t period, si means the equipment i starts maintenance in the si period, mi is the maintenance period, ei means the earliest time to start maintenance, and li means the latest time to start maintenance.


### 2.2.2. Maintenance Resource Constraints

In a maintenance cycle, the number of equipment that can be maintained at the same time is limited.(4)∑i=1Nzituit≤Zmax,where Zmax indicates the upper limit of maintenance workload in the t period.


### 2.2.3. Simultaneous Maintenance Constraints

Equipment maintenance shall avoid repeated power failure at the same load point, and all problems shall be solved during power failure maintenance.(5)sk=si,where sk is the time for equipment k to start maintenance and si is the time for equipment i to start maintenance.


### 2.2.4. Mutually Exclusive Maintenance Constraints

To prevent the expansion of power outage, some power equipment cannot be arranged for maintenance at the same time.(6)sj>si+mi+1,where sj is the time when equipment J starts maintenance; si is the time when equipment i starts maintenance; mi is the maintenance period of equipment i.


### 2.2.5. Security Constraints

In order to ensure the safe and stable operation of power grid, the maintenance plan must be checked by power flow calculation.(7)Ii≤Ii,max,(8)Vi,min≤Vi≤Vi,max,(9)Pi≤Pi,max,where Ii,max, Vi,max, Vi,min, and Pi,max are, respectively, expressed as the upper and lower limit of current flow, the upper and lower limit of node voltage, and the allowable power limit. Ii is the current flowing through equipment I, Vi is the node voltage amplitude, and Pi is the active power passing through the device.


## 2.2.1. Time Constraints

Any maintenance work shall be completed on time.(3)uit1si≤t≤si+mi,0,where uit refers to the equipment i maintenance status, uit=1 means the equipment i is powered off for maintenance in the t period, uit=0 means the equipment i is in normal operation for maintenance in the t period, si means the equipment i starts maintenance in the si period, mi is the maintenance period, ei means the earliest time to start maintenance, and li means the latest time to start maintenance.


## 2.2.2. Maintenance Resource Constraints

In a maintenance cycle, the number of equipment that can be maintained at the same time is limited.(4)∑i=1Nzituit≤Zmax,where Zmax indicates the upper limit of maintenance workload in the t period.


## 2.2.3. Simultaneous Maintenance Constraints

Equipment maintenance shall avoid repeated power failure at the same load point, and all problems shall be solved during power failure maintenance.(5)sk=si,where sk is the time for equipment k to start maintenance and si is the time for equipment i to start maintenance.


## 2.2.4. Mutually Exclusive Maintenance Constraints

To prevent the expansion of power outage, some power equipment cannot be arranged for maintenance at the same time.(6)sj>si+mi+1,where sj is the time when equipment J starts maintenance; si is the time when equipment i starts maintenance; mi is the maintenance period of equipment i.


## 2.2.5. Security Constraints

In order to ensure the safe and stable operation of power grid, the maintenance plan must be checked by power flow calculation.(7)Ii≤Ii,max,(8)Vi,min≤Vi≤Vi,max,(9)Pi≤Pi,max,where Ii,max, Vi,max, Vi,min, and Pi,max are, respectively, expressed as the upper and lower limit of current flow, the upper and lower limit of node voltage, and the allowable power limit. Ii is the current flowing through equipment I, Vi is the node voltage amplitude, and Pi is the active power passing through the device.


## 2.3. Optimization Model

In this paper, the expected value of energy shortage (EENS) is taken as the reliability evaluation index, and the maintenance cost is taken as the economic evaluation index of maintenance plan.(10)minfiX,s.t.giX=0,hjX≤0,where fiX represents the number i objective function of power equipment maintenance plan, X is the n-dimensional decision vector, m means number of objective function, giX and hjX, respectively, represents the equality constraint function and inequality constraint function, p is the number of equality constrained functions, and q is the number of inequality constrained function.In most cases, the economic objective function and reliability objective function are in conflict with each other. The improvement of one objective may lead to the performance degradation of the other, and it is impossible to make both objectives reach the optimal solution at the same time. It can only be dealt with in the middle of them to make each subobjective reach the optimization as much as possible. The solution of a multiobjective optimization problem is usually a set of solutions that meet the conditions. For the solution set, we can find a set of solutions that are as close to the best as possible.


## 3. Optimization Model Solving Method

### 3.1. DDRQN Algorithm

For complex decision-making problems, the decision-making ability of the single agent system is still far from enough. The multiobjective power equipment maintenance planning model established in this paper is competitive or cooperative in economy and reliability. Therefore, under certain conditions, the DRL algorithm needs to be extended to a multiagent system in which multiple agents cooperate or compete with each other. DDRQN allocates the deep recurrentq-networks (DRQN) training module for each agent to build a multiagent system. Its function expression is as follows:(11)Ltm=ytm−Qotm,ht−1m,m,at−1m,atm;θi2,where ytm is the target Q-value function:(12)ytm=Rt,ifst,Rt+γ⋅Qot+1m,htm,m,atm,a;θi−,where Qotm,ht−1m,m,at−1m,atm;θi is expressed as Q function, otm is the observation of agent with number m at time t, ht−1m represents the long- and short-term memory hidden layer state of agent number m at time t − 1, atm means the action corresponding to the current Q-value function of the agent number m, θi is the network weight corresponding to the agent number m in the i round iteration, θi− is the target network weight corresponding to the agent number m in the i round iteration, m is the number of the agent currently processed, at−1m is a part of the state action history sequence, γ is the discount factor, and st is the status of time t. Rt is the reward function. θi and θi− are expressed as follows:(13)θi+1=θi+α⋅∇θ,θi+1−=θi−+α−⋅θi+1−θi−,where ∇θ is the gradient value, α is the learning rate, and α− is the target learning rate.


### 3.2. Transforming Optimization Objectives into Deep Reinforcement Learning Tasks

A deep reinforcement learning problem must have two conditions: agent and environment. Its learning and decision-making process is as follows: according to agent strategy and environment information, the agent makes corresponding actions to make the reward optimal [25, 26]. Deep reinforcement learning has been successful in other areas, but the decision-making optimization of power equipment maintenance plan is not the same as other environments. Therefore, the key to its successful application is to transform the decision-making optimization problem of power equipment maintenance plan into deep reinforcement learning task.The decision-making and optimization of power equipment maintenance plan can be used as a dynamic environment, and the equipment to be overhauled can be used as theat of an agent. For any t time, the power generation and load of the system can be used as a state, and st is defined as follows:(14)st=PG,PLoad.Under the action ofat, the reward of environmental feedback can be determined by optimization objective (1). The decision-making optimization problem of power equipment maintenance plan is transformed into solving the optimal solution of the equation set. Then, Build up an action A for the power equipment which need to be overhauled. After that, calculate and get the state st and reward at of feedback,  when the action value is rt. The memory unit is shown as dt=at,st,rt. By calculating the loss error, the agent learns the strategy corresponding to the power equipment maintenance problem, namely, the maintenance plan. The solution expression is as follows:(15)Ltm=ytm−Qotm,ht−1m,m,at−1m,atm;θi2,ytm=Rt,ifst,Rt+γ⋅Qot+1m,htm,m,atm,a;θi−,Rt=fX=f1X,f2X.The agent learning process is shown in Figure1.Figure 1
Schematic diagram of agent learning process.


### 3.3. Solution Flow

The decision-making and optimization process of the power equipment maintenance plan is shown in Figure2, which is mainly divided into three parts: data preparation, algorithm solution, and safety check.Figure 2
Flow chart of decision optimization of the power grid maintenance plan.Data preparation: acquisition of initial data, including the list of equipment to be overhauled and the date of overhauling, the overhauling plan arranged by the superior, the remaining overhauling plan, and the data of power grid operation mode; establishment of the optimization model of power equipment overhauling plan based on the acquired initial data according to expression (15); adjustment of power grid operation mode based on the scheduled overhauling plan, and adjustment of the adjusted operation mode The mode of safety check is adopted to ensure that the power grid operation is safe and stable at this time. If the safety and stability conditions are not met, the alarm information will be output.Algorithm solution: DDRQN algorithm solution process is used according to the current power grid operation state to get the power generationPG and load of the system PLoad so as to get the initial state st; at this time, the power grid is safe and stable operation state; build action set A with the power equipment to be overhauled; in the process of performing an action at exploration, get the state st+1 and reward rt+1 so as to calculate the target Q-value function ytm, Q-value function, and loss error Ltm. Through the gradient descent algorithm, all parameters of the DDRQN network are updated; through one-time solution process, under the condition of equipment I maintenance, EENS and maintenance cost of power grid at this time are obtained; based on a certain strategy, under the condition of state st+1, the next exploration action at+1 is made to start the next solution process, and the solution is not terminated until all equipment to be overhauled are arranged and the arrangement results meet the constraint conditions. At this time, the strategy is the power equipment maintenance plan.Safety check: take the maintenance strategy as the section to calculate the power flow and carry out safety check. If the constraint conditions (7), (8), and (9) for safe operation of the power grid are not met, update the parameters at, st, and rt of the DDRQN algorithm and solve it again. Carry out n − 1 check for the maintenance plan and output the alarm prompt information if the condition for stable operation of the power grid is not met. After a series of iterative solutions, the optimal maintenance strategy is obtained.


## 3.1. DDRQN Algorithm

For complex decision-making problems, the decision-making ability of the single agent system is still far from enough. The multiobjective power equipment maintenance planning model established in this paper is competitive or cooperative in economy and reliability. Therefore, under certain conditions, the DRL algorithm needs to be extended to a multiagent system in which multiple agents cooperate or compete with each other. DDRQN allocates the deep recurrentq-networks (DRQN) training module for each agent to build a multiagent system. Its function expression is as follows:(11)Ltm=ytm−Qotm,ht−1m,m,at−1m,atm;θi2,where ytm is the target Q-value function:(12)ytm=Rt,ifst,Rt+γ⋅Qot+1m,htm,m,atm,a;θi−,where Qotm,ht−1m,m,at−1m,atm;θi is expressed as Q function, otm is the observation of agent with number m at time t, ht−1m represents the long- and short-term memory hidden layer state of agent number m at time t − 1, atm means the action corresponding to the current Q-value function of the agent number m, θi is the network weight corresponding to the agent number m in the i round iteration, θi− is the target network weight corresponding to the agent number m in the i round iteration, m is the number of the agent currently processed, at−1m is a part of the state action history sequence, γ is the discount factor, and st is the status of time t. Rt is the reward function. θi and θi− are expressed as follows:(13)θi+1=θi+α⋅∇θ,θi+1−=θi−+α−⋅θi+1−θi−,where ∇θ is the gradient value, α is the learning rate, and α− is the target learning rate.


## 3.2. Transforming Optimization Objectives into Deep Reinforcement Learning Tasks

A deep reinforcement learning problem must have two conditions: agent and environment. Its learning and decision-making process is as follows: according to agent strategy and environment information, the agent makes corresponding actions to make the reward optimal [25, 26]. Deep reinforcement learning has been successful in other areas, but the decision-making optimization of power equipment maintenance plan is not the same as other environments. Therefore, the key to its successful application is to transform the decision-making optimization problem of power equipment maintenance plan into deep reinforcement learning task.The decision-making and optimization of power equipment maintenance plan can be used as a dynamic environment, and the equipment to be overhauled can be used as theat of an agent. For any t time, the power generation and load of the system can be used as a state, and st is defined as follows:(14)st=PG,PLoad.Under the action ofat, the reward of environmental feedback can be determined by optimization objective (1). The decision-making optimization problem of power equipment maintenance plan is transformed into solving the optimal solution of the equation set. Then, Build up an action A for the power equipment which need to be overhauled. After that, calculate and get the state st and reward at of feedback,  when the action value is rt. The memory unit is shown as dt=at,st,rt. By calculating the loss error, the agent learns the strategy corresponding to the power equipment maintenance problem, namely, the maintenance plan. The solution expression is as follows:(15)Ltm=ytm−Qotm,ht−1m,m,at−1m,atm;θi2,ytm=Rt,ifst,Rt+γ⋅Qot+1m,htm,m,atm,a;θi−,Rt=fX=f1X,f2X.The agent learning process is shown in Figure1.Figure 1
Schematic diagram of agent learning process.


## 3.3. Solution Flow

The decision-making and optimization process of the power equipment maintenance plan is shown in Figure2, which is mainly divided into three parts: data preparation, algorithm solution, and safety check.Figure 2
Flow chart of decision optimization of the power grid maintenance plan.Data preparation: acquisition of initial data, including the list of equipment to be overhauled and the date of overhauling, the overhauling plan arranged by the superior, the remaining overhauling plan, and the data of power grid operation mode; establishment of the optimization model of power equipment overhauling plan based on the acquired initial data according to expression (15); adjustment of power grid operation mode based on the scheduled overhauling plan, and adjustment of the adjusted operation mode The mode of safety check is adopted to ensure that the power grid operation is safe and stable at this time. If the safety and stability conditions are not met, the alarm information will be output.Algorithm solution: DDRQN algorithm solution process is used according to the current power grid operation state to get the power generationPG and load of the system PLoad so as to get the initial state st; at this time, the power grid is safe and stable operation state; build action set A with the power equipment to be overhauled; in the process of performing an action at exploration, get the state st+1 and reward rt+1 so as to calculate the target Q-value function ytm, Q-value function, and loss error Ltm. Through the gradient descent algorithm, all parameters of the DDRQN network are updated; through one-time solution process, under the condition of equipment I maintenance, EENS and maintenance cost of power grid at this time are obtained; based on a certain strategy, under the condition of state st+1, the next exploration action at+1 is made to start the next solution process, and the solution is not terminated until all equipment to be overhauled are arranged and the arrangement results meet the constraint conditions. At this time, the strategy is the power equipment maintenance plan.Safety check: take the maintenance strategy as the section to calculate the power flow and carry out safety check. If the constraint conditions (7), (8), and (9) for safe operation of the power grid are not met, update the parameters at, st, and rt of the DDRQN algorithm and solve it again. Carry out n − 1 check for the maintenance plan and output the alarm prompt information if the condition for stable operation of the power grid is not met. After a series of iterative solutions, the optimal maintenance strategy is obtained.


## 4. Case Analysis

In order to verify the distributed deep cycleQ network to achieve the decision-making optimization of power equipment maintenance plan, this paper uses the IEEE-118 node system to verify its network topology as shown in Figure 3, including 54 generators and 132 lines. The verification environment is based on MATLAB R2016 with Intel Core i7-8550U CPU and 8GB RAM computer.Figure 3
Topology of the IEEE-118 node network.


### 4.1. Parameter Setting

In the process of model solving, three system parametersγ, α, and α− need to be set, which are usually empirical values. For the learning rate α, the value range is (0, 1); the larger α can speed up the learning speed, and the smaller α can improve the stability of the solution. α− is the target learning rate, which is used to update the network weight of the target value function. γ is the discount factor, which is used to discount the reward of the target value function. In this paper, we first use a wider search range and a larger step size to find the possible location of the global optimal value and then narrow the search scope and step size in order to find a more accurate optimal value.


### 4.2. Result Analysis

DDRQN algorithm, deep recurrentQ-networks algorithm, and particle swarm optimization (PSO) algorithm are used to solve the decision-making optimization model of power equipment maintenance plan established in this paper. The results are shown in Figures 4 and 5 . The abscissa represents the number of iterations, and the ordinate represents EENS (MW·h) and maintenance cost (10000 yuan). Table 1 shows the solution obtained by 100 iterations of the optimization algorithm.Figure 4
Reliability comparison of plan.Figure 5
Economic comparison of plan.Table 1
Results of different optimization algorithms.
EENS (MW·h)Maintenance cost (ten thousand yuan)PSO73.4851.06DRQN72.5550.20DDRQN71.7549.60As can be seen from Table1, the results of the optimization algorithm proposed in this paper are better than those of the other two methods because EENS and maintenance costs are less than the other two methods.It can be seen from the curve in Figure4 that the DDRQN algorithm proposed in this paper is superior to the PSO algorithm and DRQN algorithm in terms of algorithm convergence performance; the maintenance scheme obtained by DDRQN algorithm, DRQN algorithm, and PSO algorithm, respectively, is superior to the other two algorithms in terms of reliability and economy. The proposed algorithm can effectively obtain the optimal cooperative control and the quality of decision-making in the multiagent system by using its strong comprehensive decision-making ability and coordination interaction ability. Therefore, the overall performance has been greatly improved.


## 4.1. Parameter Setting

In the process of model solving, three system parametersγ, α, and α− need to be set, which are usually empirical values. For the learning rate α, the value range is (0, 1); the larger α can speed up the learning speed, and the smaller α can improve the stability of the solution. α− is the target learning rate, which is used to update the network weight of the target value function. γ is the discount factor, which is used to discount the reward of the target value function. In this paper, we first use a wider search range and a larger step size to find the possible location of the global optimal value and then narrow the search scope and step size in order to find a more accurate optimal value.


## 4.2. Result Analysis

DDRQN algorithm, deep recurrentQ-networks algorithm, and particle swarm optimization (PSO) algorithm are used to solve the decision-making optimization model of power equipment maintenance plan established in this paper. The results are shown in Figures 4 and 5 . The abscissa represents the number of iterations, and the ordinate represents EENS (MW·h) and maintenance cost (10000 yuan). Table 1 shows the solution obtained by 100 iterations of the optimization algorithm.Figure 4
Reliability comparison of plan.Figure 5
Economic comparison of plan.Table 1
Results of different optimization algorithms.
EENS (MW·h)Maintenance cost (ten thousand yuan)PSO73.4851.06DRQN72.5550.20DDRQN71.7549.60As can be seen from Table1, the results of the optimization algorithm proposed in this paper are better than those of the other two methods because EENS and maintenance costs are less than the other two methods.It can be seen from the curve in Figure4 that the DDRQN algorithm proposed in this paper is superior to the PSO algorithm and DRQN algorithm in terms of algorithm convergence performance; the maintenance scheme obtained by DDRQN algorithm, DRQN algorithm, and PSO algorithm, respectively, is superior to the other two algorithms in terms of reliability and economy. The proposed algorithm can effectively obtain the optimal cooperative control and the quality of decision-making in the multiagent system by using its strong comprehensive decision-making ability and coordination interaction ability. Therefore, the overall performance has been greatly improved.


## 5. Conclusion

In this paper, the maintenance plan optimization model of power equipment is established. The reliability evaluation index EENS is the expected value of electric energy shortage, and the maintenance cost is the economic evaluation index of the maintenance plan. The reliability and economy of power grid operation are fully considered. In this paper, a novel distributed deep loopQ network multiagent deep reinforcement learning algorithm is proposed to obtain the optimal decision of power equipment maintenance plan and improve the accuracy and intelligence level of making power equipment maintenance plan. The DDRQN algorithm uses its own comprehensive decision-making ability and coordination interaction ability to get the optimal decision among multiagent. Through case analysis, the results show that the DDRQN algorithm has strong learning ability, high adaptability, strong comprehensive decision-making ability, and better optimization performance; it can reasonably plan the maintenance plan, and the maintenance plan obtained fully takes into account the economy and reliability. The practical guiding significance of this algorithm is to realize the accuracy and automation of scheduling in the scheduling of power grid maintenance plan.



---

*Source: 9372803-2021-03-15.xml*