# Energy Management Model of Intelligent Park Based on Improved Depth Deterministic Gradient Strategy Algorithm

Authors: Chen Xu; Xianwei Jiang
Journal: Scientific Programming
 (2022)

---

## Abstract

The traditional distribution automation system, demand side management system, and distributed generation access and control system, respectively, solve the problems of regional distribution network power supply, customer power consumption, and new energy utilization to varying degrees. Intelligent Park evolved from the concept of intelligent park of State Grid Corporation, Modern enterprises, or residential areas that can make full use of modern communication, computer, automation, and other technologies to implement the power supply demand of the park. For the modern intelligent park with user-side temperature control load and demand response load access, an intelligent park energy management and optimal scheduling method based on deep reinforcement learning (DRL) algorithm is proposed. Through the interaction between the agent and the energy environment of the intelligent park, the control strategy is adaptively learned. This method can realize the continuous action control of energy management in intelligent park and can realize the optimal scheduling decision of intelligent park in various scenarios. Firstly, based on the characteristics and types of intelligent park aggregation unit, the environment model of intelligent park energy management system interacting with agent is established. Secondly, the basic principle of deep deterministic policy gradient deep algorithm is introduced, and on this basis, the key links of deep reinforcement learning algorithm, such as action space, state space, reward mechanism, neural network structure, and learning process, are designed. Finally, the effectiveness of the proposed algorithm is verified by an example of a city intelligent park.


---

## Body

## 1. Introduction

As a typical Internet of Things [1–6] application, intelligent park refers to a regional autonomous system composed of distributed photovoltaic and energy storage units on the user side and various flexibly adjusted loads, which coordinates with the distribution network or independently supplies power to the loads [7–9]. In recent years, intelligent parks have shown great potential in grid connection and consumption of distributed new energy, aggregation, and regulation of controllable load on the user side and have become one of the main strategies for developing distributed energy and improving energy utilization rate on the user side in many developed countries [10]. Different from the traditional microgrid concept, smart parks add flexible and controllable resources such as air conditioners and electric vehicles to their aggregation units, which aggravates the uncertainty and complexity of commercial transactions and power logistics in smart grids to a certain extent [11]. Therefore, how to understand personal electricity consumption behavior and its impact from a large number of high-dimensional converged data on the user side and fully mobilize the flexibility of user-side resources has become an urgent problem to be solved in microgrid energy management [12, 13].At present, for the energy management and optimal scheduling of intelligent parks, from the perspective of algorithm, most of them focus on optimization algorithm or heuristic intelligent algorithm. Sun et al. in [14] established an optimal dispatching model of intelligent park considering photovoltaic, ice storage air conditioning load, and interruptible load on the user side and established a mixed integer programming model of intelligent park. On this basis, the power flow constraint of distribution network is further considered in [15], and the energy management model of microgrid is decomposed into a unit commitment problem and an optimal power flow problem, which avoids solving the mixed integer nonlinear programming problem directly. Zhang et al. in [16] further studied the optimal scheduling problem of microgrid under uncertain photovoltaic output and proposed a two-stage robust optimization model of microgrid based on C-CG algorithm. Lei et al. in [17] studied multi-microgrid complementary scheduling and energy sharing on the basis of single microgrid operation optimization, established a master-slave game model between multi-microgrid and distribution network, and used Kriging metamodel fitting with less computation to replace the energy management model of lower microgrid. On this basis, Zhao and Tao et al. in [18, 19] further study the distributed optimization of multi-microgrid and realize distributed computing of multi-microgrid based on alternating direction multiplier method, which protects the privacy of each microgrid. Askarzadeh in [20] uses a genetic algorithm based on memory mechanism to directly solve the minimum operating cost problem of microgrid. In [21], hybrid particle swarm optimization algorithm is used to solve the multiobjective optimization problem considering both distribution network security and microgrid operation economy.The research methods of optimizing the operation of intelligent parks in the above documents can be summarized as centralized and distributed solutions. Specifically, when using these traditional methods for optimization, all possible solutions must be calculated completely or partially, and the best solution must be selected. However, this kind of calculation leads to the low computational efficiency of the traditional optimization process, and when the optimization model is nonlinear, it is easy to fall into the local optimal solution. Relying on the background of big data, deep reinforcement learning (DRL) algorithm provides a new way to solve this problem. As an artificial intelligence scheme, DRL algorithm has been widely used in the fields of optics, communication, geography, and power system scheduling. She et al. in [22] applied unsupervised deep learning and deep reinforcement learning to 6G mobile communication, which verifies the effectiveness of different algorithms. The deep Q-learning (DQN) algorithm is applied to cooperative communication technology [23], and DQN is trained according to interrupt probability and mutual information, and the optimal relay is selected from multiple relay nodes without network model or prior data. Cao et al. in [24] used the deep reinforcement learning algorithm to optimize the arbitrage strategy of energy storage system and forecasted the load data by combining convolution neural network and long-term memory. Li et al. and Qiu et al. in [25, 26] further applied deep reinforcement learning to the charge and discharge optimization of electric vehicles.As a model-free optimization algorithm, DRL has been widely used in the fields of demand response optimization, power market decision optimization, and power system optimal dispatching. Wang et al. in [27] designed a model-free deep reinforcement learning method with double DQN structure to optimize the demand response management under time-of-use electricity price and variable power consumption mode; the actor-critical algorithm is used to study the demand response regulation of load aggregation quotient, and the effectiveness of the proposed algorithm is proved under the condition of considering various uncertain factors [28]; Ye et al. in [29] studied the optimization of electricity market clearing and used deep deterministic policy gradient (DDPG) algorithm to solve the bilevel model. Liu et al. in [30] used A3C algorithm to realize residents’ load participating in demand side response and made use of CPU multithreading function to execute multiple actions at the same time through multiagent. It can be seen that DRL can learn from the continuous transformation of historical data. This powerful machine learning model, combined with deep neural network, can quickly extract, control, and optimize the terminal aggregation units of intelligent park, and can deal with the high uncertainty under various complex modes. The DRL algorithm has been applied to various fields of power system from different angles in the literature mentioned above, but in general, there are still some shortcomings as follows: (1) Although the existing DRL algorithm can improve the convergence by designing the target network, experience playback, and other mechanisms, its training is slow and converges. Difficulties, instability, and other problems are still widespread, and it is difficult to adjust parameters; (2) although some researches have applied DRL algorithm to the cooptimal scheduling of the park, the park model is simple and often ignores the user-side demand response resources, such as temperature-controlled load, transferable/interruptible load, etc., so it is difficult to reflect the benign interaction of source-load. Based on the above considerations, this paper improves the existing DDPG algorithm from the perspectives of random exploration strategy, environment awareness and learning ratio, and increasing experience playback pool and applies the improved DDPG algorithm to the energy management and optimal scheduling decision of modern intelligent parks. Finally, an example of an actual intelligent park in a city is given to verify the effectiveness of the proposed algorithm in different scenarios.To sum up, the main contributions of this paper are as follows:(1)
Aggregate modeling of user-side multishape flexible regulation resources in intelligent park, construct user-side temperature control load model and price demand response model, and take user-side regulation into account in the energy management model of intelligent park.(2)
An energy management model based on improved depth deterministic gradient strategy is proposed in this paper. Through such three measures as the random exploration strategy, the adjustment of environment perception and learning proportion, and the increase of high access experience pool, the convergence characteristics and computing ability of the algorithm are improved significantly.(3)
The effectiveness of the proposed algorithm is verified by an example of an actual intelligent park in a city, and the improvement of the overall economic benefit of the park by controlling user-side resources is verified by a comparative example.


## 2. Intelligent Park Structure and Scheduling Framework

The microgrid studied in this paper is a community microgrid with independent supply and demand infrastructure. The microgrid is managed by an energy aggregator or a public utility company. There are wind turbines in the microgrid, which are connected to the main power grid and can supply internal load or sell electricity to the superior power grid according to the market electricity price. The overall architecture of the intelligent park is shown in Figure1, its architecture can be analyzed from three angles: physical layer, information layer, and control layer. The physical layer includes a wind turbine, an electric energy storage system, a set of temperature-controlled loads, and a set of price-based demand response loads. The information layer is mainly composed of smart meters and communication systems, which can realize real-time monitoring of power generation and consumption data and bidirectional transmission of information of various components. The control layer is mainly an energy management system (EMS) that sends control signals to the controllable components of the microgrid through the relevant infrastructure. As can be seen from Figure 1, the main control variables include (1) switch control quantity of temperature control load; (2) charge/discharge control of electric energy storage system; (3) interactive power control with superior power grid. Therefore, when modeling energy management in this paper, a multiagent control system can be established according to these control variables. These control units will be modeled one by one below.Figure 1
Overall structure of community micronetwork.


### 2.1. Modeling of Electric Energy Storage System

Considering the economic and technical feasibility, this paper adopts a community shared energy storage system, which can meet the power demand of intelligent park for at least two hours. The real-time stored power status of the energy storage system is expressed as follows:(1)Bt=Bt−1+ηcCt−Dtηd,where Bt∈0,Bmax is the electric energy stored by the electric energy storage system at time t, Bmax is the maximum capacity of the electric energy storage system, ηc and ηd, respectively, represent the charging efficiency and discharging efficiency of the energy storage system, Ct∈0,Cmax and Dt∈0,Dmax, respectively, represent the charging power and discharging power of ESS at time t, and Cmax and Dmax, respectively, represent the maximum charging power and discharging power of ESS. The storage state of the ESS system can be expressed by SOC values as(2)BSCt=BtBmax.


### 2.2. Temperature Control Load Modeling

The temperature control load in the intelligent park refers to a large number of air conditioners, water heaters, heat pumps, or refrigerators, which can form flexible adjustment resources with a certain controllable capacity through aggregators and can participate in the resource scheduling of the intelligent park as a whole under the coordinated control of agents. Its basic control structure is shown in Figure2.Figure 2
TCL aggregation control chart.In order to ensure the comfort level of users, a feedback controller is equipped on each TCL to maintain the temperature of related equipment within an acceptable range. The feedback controller can receive the switch action information of the equipment from the aggregator and verify whether the temperature constraint can be met under the action. The action control strategy of the feedback controller is as follows:(3)ut+1i=0ifTti>TmaxiutiifTmaxi>Tti>Tmini1ifTti<Tmini,where uti is the decision variable of the feedback controller, indicating the off-state of the equipment; if it is in the open state, it is set to 1; otherwise it is set to 0; Tti represents the temperature of the i-th temperature control load at time t, and Tmini and Tmaxi are the minimum and maximum values of temperature, respectively. The temperature dynamic change process of temperature control load satisfies the second law of thermodynamics [31], which is specifically expressed as follows:(4)Tt+1i=1CaiTt0−Tti+1CmiTm,ti−Tti+Ltcliuti+qi.Tm,t+1i=1CmiTti−Tm,ti,where Tti is the measured indoor air temperature, Tm,ti is the unobservable building temperature, Tt0 is the outdoor temperature, Cai and Cai are the specific heat capacity of air and buildings, qi is the internal heat of buildings, and Ltcli is the rated power of temperature control load. Similarly, the adjustable range of temperature control load can also be expressed in the form of state of charge, specifically as follows:(5)SoCti=Tti−TminiTmaxi−Tmini.


### 2.3. Price-Based Demand Response Load Modeling

Price-based demand response load refers to some interruptible/transferable loads in intelligent parks, which can participate in the energy scheduling of microgrid on the premise of meeting the minimum demand of users according to the electricity price. Based on sensitivity coefficientβ∈0,1 and recovery coefficient λi, this paper describes the characteristics of demand response load of different households. The so-called sensitivity coefficient refers to the load percentage that response load I can be reduced/increased under the condition of given electricity price increase/decrease; the so-called recovery coefficient refers to how many times the reduced load can be supplemented. The specific load size of price demand response load I at time t is(6)Lti=Lb,t−SLti+PBti.SLti=Lb,t⋅βi⋅δt.In the formula,Lb,t tables the basic load size, SLti represents the actual load transferred at time t, the specific expression is shown in (6), δt=−2,−1,0,1,2 represents the load electricity price level at time t, δt>0 at that time, and the electricity price is higher at that time. EMS reduces the operating cost by transferring the load, so SLti>0 at this time; similarly, δt<0 at that time and SLti<0 at this time. PBti represents the compensation amount of the previously transferred load at time t, and its specific calculation method is as follows:(7)PBti=∑j=0t−1wi,jSLji,where wi,j∈0,1 represents whether load transfer occurs at time, and this variable is a Boolean variable, which is mainly affected by electricity price level and recovery coefficient. Specifically, when the transfer duration is closer to the maximum recovery coefficient, the greater probability of wi,j=1, and the actual probability of wi,j=1 at time t can be carried out in the following way:(8)Pwi,j=1=clip−δt⋅signSLjt2+t−jλi,0,1.clipX,a,b=aifX<aXifa≤X≤bbifX>b.The intelligent park is also connected with the superior power grid through tie lines to balance the internal supply and demand relationship in real time. That is to say, when the internal supply and demand relationship cannot be balanced, the superior power grid can be used as the standby dispatching resource of the microgrid, and its transaction price adopts the Finnish electricity market price provided by [32]. The agent of the main power grid needs to share the purchase/sale price information with EMS in real time, which is recorded as λtu,λtd. The agent control flow is shown in Figure 3, and the specific steps are as follows:(1)
According to the electricity price level, determine the basic electricity consumption strategy of demand response load and temperature control load.(2)
Determine the relationship between energy supply and demand according to the demand response, the actual electricity consumption of temperature-controlled load, and the output of wind turbines.(3)
According to the relationship between energy supply and demand, priority is given to scheduling energy storage to reduce power imbalance, and if the supply is in short supply, energy storage will discharge. If the supply exceeds the demand, the excess energy will be used for energy storage and charging.(4)
If the energy balance cannot be realized after (3), the purchase/sale of electricity from the superior power grid shall be considered, and the purchase/sale value shall be recorded asPtbuy,Ptsell until the energy balance is realized.Figure 3
Basic control structure of EMS.EMS should fully consider the price response characteristics of various resources and determine the load electricity price levelδt at all times in the intelligent park, and in order to prevent the actual owner of the park from maliciously raising the electricity price, it is stipulated that the quotation of EMS should fluctuate around an intermediate value, and at the same time, the daily average price level should be close to the retail electricity price provided by power retailers. See the following formula for the specific form of EMS quotation:(9)Pt∈Pmarket+δt⋅cstδt∈−2,−1,0,1,2.In the formula, in order to reflect a constant of price change speed,Pmarket is the retail price. In the actual operation process, EMS continuously records the price level for each short time and accumulates it. When the price level exceeds a certain threshold, the retail price is used as the actual transaction price. The final price level calculation formula is as follows:(10)δt,eff=δtif∑j=0tδt≤threshold0if∑j=0tδt≤threshold,where threshold is the preset threshold.


## 2.1. Modeling of Electric Energy Storage System

Considering the economic and technical feasibility, this paper adopts a community shared energy storage system, which can meet the power demand of intelligent park for at least two hours. The real-time stored power status of the energy storage system is expressed as follows:(1)Bt=Bt−1+ηcCt−Dtηd,where Bt∈0,Bmax is the electric energy stored by the electric energy storage system at time t, Bmax is the maximum capacity of the electric energy storage system, ηc and ηd, respectively, represent the charging efficiency and discharging efficiency of the energy storage system, Ct∈0,Cmax and Dt∈0,Dmax, respectively, represent the charging power and discharging power of ESS at time t, and Cmax and Dmax, respectively, represent the maximum charging power and discharging power of ESS. The storage state of the ESS system can be expressed by SOC values as(2)BSCt=BtBmax.


## 2.2. Temperature Control Load Modeling

The temperature control load in the intelligent park refers to a large number of air conditioners, water heaters, heat pumps, or refrigerators, which can form flexible adjustment resources with a certain controllable capacity through aggregators and can participate in the resource scheduling of the intelligent park as a whole under the coordinated control of agents. Its basic control structure is shown in Figure2.Figure 2
TCL aggregation control chart.In order to ensure the comfort level of users, a feedback controller is equipped on each TCL to maintain the temperature of related equipment within an acceptable range. The feedback controller can receive the switch action information of the equipment from the aggregator and verify whether the temperature constraint can be met under the action. The action control strategy of the feedback controller is as follows:(3)ut+1i=0ifTti>TmaxiutiifTmaxi>Tti>Tmini1ifTti<Tmini,where uti is the decision variable of the feedback controller, indicating the off-state of the equipment; if it is in the open state, it is set to 1; otherwise it is set to 0; Tti represents the temperature of the i-th temperature control load at time t, and Tmini and Tmaxi are the minimum and maximum values of temperature, respectively. The temperature dynamic change process of temperature control load satisfies the second law of thermodynamics [31], which is specifically expressed as follows:(4)Tt+1i=1CaiTt0−Tti+1CmiTm,ti−Tti+Ltcliuti+qi.Tm,t+1i=1CmiTti−Tm,ti,where Tti is the measured indoor air temperature, Tm,ti is the unobservable building temperature, Tt0 is the outdoor temperature, Cai and Cai are the specific heat capacity of air and buildings, qi is the internal heat of buildings, and Ltcli is the rated power of temperature control load. Similarly, the adjustable range of temperature control load can also be expressed in the form of state of charge, specifically as follows:(5)SoCti=Tti−TminiTmaxi−Tmini.


## 2.3. Price-Based Demand Response Load Modeling

Price-based demand response load refers to some interruptible/transferable loads in intelligent parks, which can participate in the energy scheduling of microgrid on the premise of meeting the minimum demand of users according to the electricity price. Based on sensitivity coefficientβ∈0,1 and recovery coefficient λi, this paper describes the characteristics of demand response load of different households. The so-called sensitivity coefficient refers to the load percentage that response load I can be reduced/increased under the condition of given electricity price increase/decrease; the so-called recovery coefficient refers to how many times the reduced load can be supplemented. The specific load size of price demand response load I at time t is(6)Lti=Lb,t−SLti+PBti.SLti=Lb,t⋅βi⋅δt.In the formula,Lb,t tables the basic load size, SLti represents the actual load transferred at time t, the specific expression is shown in (6), δt=−2,−1,0,1,2 represents the load electricity price level at time t, δt>0 at that time, and the electricity price is higher at that time. EMS reduces the operating cost by transferring the load, so SLti>0 at this time; similarly, δt<0 at that time and SLti<0 at this time. PBti represents the compensation amount of the previously transferred load at time t, and its specific calculation method is as follows:(7)PBti=∑j=0t−1wi,jSLji,where wi,j∈0,1 represents whether load transfer occurs at time, and this variable is a Boolean variable, which is mainly affected by electricity price level and recovery coefficient. Specifically, when the transfer duration is closer to the maximum recovery coefficient, the greater probability of wi,j=1, and the actual probability of wi,j=1 at time t can be carried out in the following way:(8)Pwi,j=1=clip−δt⋅signSLjt2+t−jλi,0,1.clipX,a,b=aifX<aXifa≤X≤bbifX>b.The intelligent park is also connected with the superior power grid through tie lines to balance the internal supply and demand relationship in real time. That is to say, when the internal supply and demand relationship cannot be balanced, the superior power grid can be used as the standby dispatching resource of the microgrid, and its transaction price adopts the Finnish electricity market price provided by [32]. The agent of the main power grid needs to share the purchase/sale price information with EMS in real time, which is recorded as λtu,λtd. The agent control flow is shown in Figure 3, and the specific steps are as follows:(1)
According to the electricity price level, determine the basic electricity consumption strategy of demand response load and temperature control load.(2)
Determine the relationship between energy supply and demand according to the demand response, the actual electricity consumption of temperature-controlled load, and the output of wind turbines.(3)
According to the relationship between energy supply and demand, priority is given to scheduling energy storage to reduce power imbalance, and if the supply is in short supply, energy storage will discharge. If the supply exceeds the demand, the excess energy will be used for energy storage and charging.(4)
If the energy balance cannot be realized after (3), the purchase/sale of electricity from the superior power grid shall be considered, and the purchase/sale value shall be recorded asPtbuy,Ptsell until the energy balance is realized.Figure 3
Basic control structure of EMS.EMS should fully consider the price response characteristics of various resources and determine the load electricity price levelδt at all times in the intelligent park, and in order to prevent the actual owner of the park from maliciously raising the electricity price, it is stipulated that the quotation of EMS should fluctuate around an intermediate value, and at the same time, the daily average price level should be close to the retail electricity price provided by power retailers. See the following formula for the specific form of EMS quotation:(9)Pt∈Pmarket+δt⋅cstδt∈−2,−1,0,1,2.In the formula, in order to reflect a constant of price change speed,Pmarket is the retail price. In the actual operation process, EMS continuously records the price level for each short time and accumulates it. When the price level exceeds a certain threshold, the retail price is used as the actual transaction price. The final price level calculation formula is as follows:(10)δt,eff=δtif∑j=0tδt≤threshold0if∑j=0tδt≤threshold,where threshold is the preset threshold.


## 3. Reinforcement Learning Framework for Energy Management in Intelligent Parks

The energy management problem of intelligent park is essentially a sequential decision-making problem, and its mathematical essence is stochastic dynamic programming. At present, most of the common methods to solve this kind of problems are based on fine model, which depend on various numerical solutions or heuristic intelligent acid and often require the objective function to be derivative or differentiable. The reinforcement learning algorithm adopted in this paper is a data-driven model-free algorithm. Adaptive learning is carried out by means of “trial and error” of agents. Through continuous interaction with the environment, constantly obtain the environmental state and take corresponding actions to change the environmental state according to certain updating strategies. In this process, the agent will get certain rewards or punishments, and the agent will use the reward value/punishment value as the updating guidance of model parameters, in order to get the maximum cumulative rewards in the process of continuous learning.


### 3.1. Markov Decision Process

The above-mentioned process of perception-action-evaluation-learning is also called Markov decision process (MDP). The four basic elements of MDP are state space, action space, transition function, and reward function. Among them, transition function describes the probability of agent changing from states∈S to s′∈S under a given action:(11)T:SAS⟶0,1Ts,a,s′=pst+1=s′|st=s,at=a.At every moment, the agent selects the corresponding action in the possible action setA according to the state st and the random strategy π (the strategy π is a function of mapping the state st to the action a). In return, the agent receives the understood reward value Rt of the process and updates it to the next state variable st+1. The ultimate goal of the agent under the policy π is to maximize the accumulated discount reward value, which is expressed as follows:(12)ρt=∑k=tD−1γk−tRk.In the formula,γ∈0,1 is the discount factor, which directly determines the influence degree of rewards in subsequent steps on the current period. According to the strategy π, the expected value of rewards under the current strategy can be obtained as follows:(13)Vπst=EπRt+γVπSt+1,∀t<D−1.The Q value function is the action value function, which indicates the expected return after selecting the actionat:(14)Qπst,at=Rt+γVπst+1.Starting from the states0, the agent begins to search for the best strategy step by step, which maximizes the action value function:(15)π∗=argmaxπQπs,a,(16)Qπ=∗s,a=Qπ∗s,a.From equations (14)–(19), it can be seen that the model-free reinforcement learning method does not need to know the specific expression of the state transition probability function T but only needs to train according to the transition process of each interaction with the environment, so that the expected value of the total reward can reach the highest. The energy management and optimal dispatching method of power grid is transformed into Markov decision model. Then the rule-based dynamic energy management method is selected as the basic strategy, which can reduce the overall optimization cost, so as to avoid solving the optimal value function of all states. Then the adaptive learning control strategy is proposed, which can realize the continuous action control of energy management in intelligent park and the optimal dispatching decision in various scenarios.


### 3.2. Reinforcement Learning Framework for Energy Management in Intelligent Parks

In this paper, the intelligent park system is the agent’s environment. The agent optimizes the scheduling by adjusting the output and power generation of related aggregation units in the system. In timeT, the environment provides the observed system state st∈S to the agent, and the agent generates action DD based on strategy st∈S and intelligent park system state at∈A.


#### 3.2.1. State Space

The state space is composed of the reference information of the decision-making of the agent at each moment, which mainly includes controllable state variableSC and external state variable SX. Among them, the controllable state variables are mainly environmental variables that can be directly/indirectly controlled by agents, such as the average SOC value of temperature-controlled load, the maximum SOC value of ESS, and the load electricity price level δt. External state variables are variables beyond the control of agents, such as temperature Tt and wind turbine power generation and higher-level market purchase and sale price Gt and Ptu,Ptd. Therefore, the state space can be described as(17)st∈S=SCxSXst=SoCt,BSCt,Gt,λtu,λtd,Lb,t,t.


#### 3.2.2. Action Space

The action space of the agent is mainly composed of four parts, including temperature control load action spaceAtcl, load price action space AP, energy storage charge and discharge action space AD, and market purchase and sale electricity action space AE, which are specifically expressed as follows:(18)at∈A=AtclxApxADxAEat=Pt,ut+1i,Ct,Dt,Ptbuy,Ptsell.


#### 3.2.3. Reward Function

The reward function is set to maximize the operational benefits of the intelligent park. The calculation method is to subtract the power generation cost from the income from selling electricity to the external power grid and the cost of purchasing electricity from the external power grid. The expression of the reward function is as follows:(19)Rt=St−Ct.St=Pt∑loadsLti+Cgen∑tclsLtcliuti+λtdPtsell.Costst=CgenGt+λtu+CtrimpPtbuy+CtrexpPtsell,where λtu,λtd are lower standby and upper standby electricity prices, which are the transaction electricity prices between microgrid and superior power grid. Gt is the power generation of wind turbines, Cgen is the power generation cost, and Ctrimp and Ctrexp are the energy transmission cost of purchasing electricity from or selling electricity to the power grid.


### 3.3. Energy Management Framework of Intelligent Park Based on DDPG Algorithm

The traditional reinforcement learning algorithm based onQ-learning performs well in dealing with small-scale discrete space problems. However, when dealing with practical problems with continuous state variables, with the increase of space dimensions, the discrete state variables will increase exponentially, which brings serious dimension disaster and cannot realize efficient learning of agents. In the energy management of intelligent park in this paper, firstly, the state variables such as wind turbine output, user load value, and market trading electricity are all continuous variables. In addition, the market transaction electricity quantity and energy storage charging and discharging power in the action space are also continuous variables, so the traditional reinforcement learning algorithm cannot effectively solve the energy management problem of intelligent park.Based on the above considerations, this paper uses deep neural network (DNN) to approximate reinforcement learning, so as to solve the energy management problem of intelligent park with continuous state/action space [33]. The DDPG algorithm based on the actor-critical framework is adopted in this paper. It uses two independent networks to approximate the critical function θQ and the actor function θQ, and each network has its own target networks θQ′ and θπ‘, for which the target Q value and target strategy are denoted as Q′ and π′.


#### 3.3.1. Value Network Training

For value networks, the network parameters are usually optimized by minimizing loss function:(20)LθQ=Eyt−Qst,at|θQ2,where yt is the target Q value; the calculation method is as follows:(21)yt=rt+γQ′st+1,π′st+1|θπ′|θQ′.The value network parameters are updated in the gradient direction, andLθQ is calculated as follows about θQ gradient:(22)∇θQLθQ=E2yt−Qst,at|θQ∇θQQst,at.According to the gradient rule, the update formula of the value network parameters is as follows:(23)θQ←θQ−μQ∇θQLθQ.


#### 3.3.2. Strategic Network Training

The training and updating of the strategy network are also based on gradient information. The gradient information of the strategy network is recorded as∇aQst,at|θQ, and the sampling strategy gradient is calculated as follows:(24)∇θππ=∇aQs,a|θQ|s=st,a=πst∇θππs,|θπ|s=st.According to the above deterministic policy gradient, the policy network parameters can be updated:(25)θπ←θπ−μπ∇θππ,where μπ is the strategic network learning rate.The target network parametersθQ′ and θπ′ sampling soft update technology further improve the stability of the learning process:(26)θQ′←τθQ−1−τθQ′.θπ′←τθπ−1−τθπ′,where τ is the soft renewal coefficient.


### 3.4. Improved DDPG Algorithm

#### 3.4.1. Improvement of Convergence Mechanism

(1) Random Exploration. In this paper, random noise is superimposed on the output actions to explore the environment; that is, in the process of training, the actions taken every time are improved as follows:(27)a^t=pat+1−pn,where a^t is the actual action after updating, and n is the noise that obeys the truncated normal distribution within the range of [−1, 1]; p is the proportion of neural network output action value at. The larger the value, the lower the randomness of exploration. In order to avoid any exploration at all, the upper limit of its value is 0.95.(2). Adjust the Proportion of Environmental Perception and Learning. In the traditional DDPG algorithm, every time an agent interacts with the environment, it needs to learn the model parameters once. In fact, this frequent learning time greatly increases the training time. Without sufficient interaction and exploration of the environment, frequent learning can easily make the agent fall into the local optimal value. In this paper, we adjust this and set the agent to learn every 30 times when it interacts with the environment.


#### 3.4.2. Adding a Playback Pool for High Access Experience

Due to the high proportion of superimposed exploration in the output actions in the early stage of model training, there are few transfer strategies with higher total return value, and the model training speed is slow, and the influence of random exploration is even higher than that of scheduling sequence itself. In order to improve the convergence speed of the algorithm, an experience playback pool for storing high-quality transfer process is added, and its admission condition is set to be higher than the average value of the total reward value in each training cycle, so as to accelerate the convergence speed of the previous algorithm.In fact, the three measures basically do not change the complexity of the algorithm from the mathematical model, only by adjusting the relevant parameters, or some of the results have been trained to optimize, because the mathematical model does not increase the complexity of the algorithm.In summary, the basic framework of solving energy management problems in intelligent parks based on improved DDPG algorithm is shown in Figure4. Under this framework, the input is state variables, including market electricity price information, wind turbine output, and other state variables, and the output is action vectors, which mainly include action variables such as charging and discharging power of electric energy storage system, power consumed by TCL, demand response load, and market purchase and sales power.Figure 4
Structure diagram of community microgrid optimal scheduling model based on improved A3C algorithm.(3). Example Test and Result Analysis. In this paper, a toolkit named Gym of Open AI is used to build the simulation environment of intelligent park. TensorFlow 1.1.4 Toolkit is used to train neural network, and the training data of wind power and market electricity price are taken from [34].In terms of model parameter setting, the charging and discharging efficiency of ESS is 0.9, the maximum charging and discharging power is set to 250 kW, the capacity is set to 500kWh, the wind power generation cost is 32/MW, the upper standby price and the lower standby price are taken from [35], Ctrimp and Ctrexp are 0.9/MW and 1.3/MW, respectively, and TCL parameters are taken from [36]. The total quantity is 100, the total period of a day is 24 h, and the retail electricity price is 5.48 euro/kW. All experimental results are average by multiple times.The structure of DDPG network is shown in Appendix in Figures5 and 6. Considering the dimensions of state space and action space, the number of hidden layer neurons of actor network is 300/100/100, and the number of hidden layer S1, A1, H1, H2 neurons of critic network is 200, 50, 100, and 100, respectively. ReLU function is used as the activation function of hidden layer. The learning rate of actor network and critic network is 0.001 and 0.0001, respectively, the learning rate of target network is 0.001, and the capacity of experience pool is 105.Figure 5
Structure of critic network and its target network.Figure 6
Structure of actor network and its target network.


### 3.5. Comparison of Convergence Speed and Learning Effect of Different Algorithms

In order to show the effectiveness of the improved DDPG algorithm proposed in this paper, the following basic comparative examples are set, as shown in Table1.Table 1
Final reward of each agent in different cases.
OptionRandomPerceptual learning ratioExperience poolsReward valueTraining time (h)1×110.16221.62√110.19419.543√30 : 110.22613.654√30 : 120.2312.16As can be seen from the table, compared with scheme 1 and scheme 2, it can be seen that the reward value is increased by about 19.75% and the training time is reduced by nearly 2 hours after adopting the random exploration strategy proposed in this paper. In addition, comparing scheme 3 and scheme 2, set the ratio of environment awareness and agent learning to 30: After 1, not only has the reward value of the agent been greatly increased, but also the training time has been obviously reduced, which is mainly due to the improvement of environment perception and agent learning ratio, and the reduction of a large number of unnecessary learning times, so the convergence speed of the algorithm is significantly improved, and the agent is prevented from falling into local optimal solution. Finally, compared with scheme 4 and scheme 3, it can be seen that, after adding experience pool, the algorithm further improves the convergence speed and reduces the training time on the premise of ensuring a high reward value. The reward curves under scenarios 1, 2, 3, and 4 are shown in Figure7.Figure 7
Reward curves in training process of different algorithms.Figure7 shows the reward curves of the four different schemes in Table 1, with different scheme configurations resulting in different curve trends. Under the random exploration strategy, further refine the final reward value of agents under different perceptual learning ratios, as shown in Figure 8. As can be seen from the figure, although the perceptual learning ratio can greatly reduce the training time of the model, avoid the model falling into local optimum, but when the training is carried out to a certain extent, too high proportion of perceptual exploration will lose more key information. As a result, the overall reward value is reduced and the training time is increased. Under the four comparison schemes, when the perceptual learning ratio is 30 : 1, the training time and the final reward value are optimal. Therefore, in the process of adjusting the parameters of the time model, it is necessary to set the perceptual learning ratio more in line with the model training in combination with practical problems.Figure 8
Relationship between training time and final reward value.In order to demonstrate the effectiveness of the improved DDPG algorithm compared with the existing one, the experimental results of the existing algorithms compared with the DQN algorithm, the actor-critic algorithm, and the PPO algorithm under different hyperparameters are tested respectively, as shown in Table2.Table 2
Experimental results of each algorithm under different super parameters.
Greedy coefficient (E)εDQNActor-criticPPODDPG1.0–40.0910.052−0.1520.1915.0–50.1230.075−0.1220.2451.0–50.1340.062−0.0910.2175.0–60.1390.109−0.0740.203As can be seen from the table, under different greedy coefficients, the final reward values of each algorithm are also different, showing different trends. The PPO algorithm has the worst adaptability to the model, and the overall learning effect is the worst. The DQN algorithm is better than actor-critic algorithm, but it lags behind the proposed algorithm. In addition, with the decreasing of the greedy coefficient, the learning effect of each algorithm is improved. Actor-critic algorithm and DDPG algorithm need the appropriate greedy coefficient to achieve the optimal learning effect.From Table3, we can see that the algorithms exhibit different performance with the change of exploring coefficient in the strategy of random exploration. On the whole, each algorithm can use the better learning effect between 0.5 and 0.75; the higher the exploring coefficient, the lower the randomness of the searching algorithm, and it is easy to fall into the local convergence, leading to insufficient learning, poor learning effect, and low exploring coefficient, which will lead to too random and frequent exploration, not only seriously affecting the convergence time of the algorithm, but also leading to insufficient key information learning and poor learning effect.Table 3
Experimental results of each algorithm under different super parameters.
Exploring coefficientPDQNActor-criticPPODDPG0.05—0.043−0.9220.1530.25—−0.024−0.0890.1970.5—0.075−0.9220.2450.75—0.062−0.9220.2130.95—0.014−0.9580.128In terms of training time and complexity of the algorithm, when setting the same hyperparameter, the comparison results of the four algorithms are shown in Table4.Table 4
Experimental results of each algorithm under different super parameters.
AttributeDQNActor-criticPPODDPGReward value0.1340.062−0.0910.217Training time (h)20.7218.1915.3712.16From the point of view of training complexity, the improved DDPG algorithm has a fast training speed due to the double-sample pool and reasonable environment perception-learning ratio and can keep good learning effect under the fast training speed.


### 3.6. Comparison of Test Results

Fix the trained neural network weights, and select the actual data of a typical day in winter for testing. See Figures9–11 for the temperature, basic load, market electricity price information, and wind power output value of that day, and set the following comparative examples.Figure 9
Load curve and temperature curve.Figure 10
Wind power output result.Figure 11
Electricity purchase price and electricity sale price curve.Case 1:.
Genetic algorithm is used to test the typical daily data; Case 2: The traditional DDPG algorithm is used to test the typical daily data (the algorithm corresponds to scheme 1 in Section3.1); Case 3: The improved DDPG algorithm proposed in this paper is used to test the typical day (algorithm comparison, Section 3.1, scheme 4).
In the above examples, the result obtained by Case 1 can be considered as the optimal control result. The total economic benefits of intelligent parks and the calculation time of algorithms under different examples are shown in Table5.
As can be seen from Table5, under three examples, the total income of the park obtained by genetic algorithm is the highest, but at the same time, its calculation time is also the longest. The final benefit values of the two schemes using deep reinforcement learning algorithm are quite different. After training based on the traditional DDPG algorithm, the final test effect is poor, and the total income value is only 339.19, far lower than the genetic algorithm and the improved DDPG algorithm proposed in this paper. In addition, it is worth noting that deep reinforcement learning is essentially a model-free algorithm. It can learn decision-making experience from massive historical data and continuously improve the decision-making ability of agents, although the final test results are not as good as the traditional model-based optimization algorithm. However, the improved DDPG algorithm can greatly reduce the calculation time and even realize the decision-making ability of millisecond response, which cannot be realized by the traditional model-based solution algorithm, on the premise that the calculation results are not much different from the optimal control algorithm. In order to prove this conclusion, under the same test environment, the scale of solving the problem is further expanded. The comparison of computing time of three algorithms and the other DRL methods is shown as Table 6.
It can be seen that, with the continuous expansion of the scheduling cycle, the difference in solution time of the three algorithms is more obvious. Because of the offline learning and online decision-making ability of deep reinforcement learning, the calculation time basically increases linearly with the scheduling cycle, while the calculation time of traditional intelligent algorithms increases exponentially with the expansion of the scheduling cycle. With the increasing number of distributed new energy and controllable load connected to the intelligent park in the future, the overall dispatching scale of the park will gradually expand in the future, and the advantages of deep reinforcement learning algorithm will be further reflected at this time.
In addition, by comparing the test results of different algorithms, under different scheduling cycles, the target function values are shown respectively in Table7.
As can be seen from the table, the test results of each algorithm are basically the same as the training results. The improved DDPG algorithm has the highest profit value in all the test algorithms because of its good learning effect in the training stage, and the PPO algorithm also has the lowest profit value in the test stage because of its poor learning effect.
Figure12 shows the optimization results of interactive power consumption between intelligent park and superior power grid under different examples. It can be seen from the figure that, under the problem framework of this paper, EMS system chooses to sell electricity to the superior power grid in 4–10 h when the market electricity price is higher and purchases electricity from the superior power grid in 16–22 h when the market electricity price is lower, which not only meets its aggregate load resource electricity demand, but also maximizes its operating income. In addition, although both the traditional DDPG algorithm and the improved DDPG algorithm can keep the same change level with the optimal results, the improved DDPG algorithm is obviously closer to the optimal results in numerical results, and the control effect is better than the basic DDPG algorithm.
In addition, in order to reflect the impact of load side regulation on the overall income of the park, the following comparative examples are set up, respectively:Case 1: consider regulating price-based demand response resources and temperature-controlled load resources simultaneouslyCase 2: only consider regulating price-based demand response resourcesCase 3: only consider regulating temperature-controlled load resourcesCase 4: do not consider any load side resources
Under four scenarios, the results of revenue and total transactions with the grid under the 24-hour scheduling cycle are shown in Table8.
From the results, it can be concluded that considering user-side resources in the form of demand response can effectively reduce the operation cost of the park. By responding positively in the period of higher electricity price and buying a large amount of electricity in the period of low electricity price to store energy for temperature-controlled load, the load curve of the park can be improved greatly while improving the economic benefit of the park. The specific equivalent load curve is shown in Figure13.Table 5
Total benefit of the park and the calculation time of the algorithm in different cases.
ParameterCase 1Case 2Case 3Profit (€)428.35339.19413.26Computing time (s)5.890.0650.039Table 6
Calculation time of three algorithms under different scheduling period.
Scheduling period (h)Case 1Case 2Case 3DQNPPO245.890.0650.0390.0410.038168216.560.5360.1560.1760.1647201967.252.681.5641.4791.625Table 7
Profit result of three algorithms under different scheduling period.
Scheduling period (h)DQNActor-criticPPODDPG24358.24319.27148.87413.261682301.872008.791374.432668.8272010820.179776.327876.9112208.32Figure 12
The results of market trading electricity under different algorithms.Table 8
Test result of three algorithms under different scheduling period.
Scheduling periodCase 1Case 2Case 3Case 4Profit (€)413.26379.27356.99298.76Trading electricity (kW)2601.892508.661887.381765.62Figure 13
Equivalent load curve under different examples.


## 3.1. Markov Decision Process

The above-mentioned process of perception-action-evaluation-learning is also called Markov decision process (MDP). The four basic elements of MDP are state space, action space, transition function, and reward function. Among them, transition function describes the probability of agent changing from states∈S to s′∈S under a given action:(11)T:SAS⟶0,1Ts,a,s′=pst+1=s′|st=s,at=a.At every moment, the agent selects the corresponding action in the possible action setA according to the state st and the random strategy π (the strategy π is a function of mapping the state st to the action a). In return, the agent receives the understood reward value Rt of the process and updates it to the next state variable st+1. The ultimate goal of the agent under the policy π is to maximize the accumulated discount reward value, which is expressed as follows:(12)ρt=∑k=tD−1γk−tRk.In the formula,γ∈0,1 is the discount factor, which directly determines the influence degree of rewards in subsequent steps on the current period. According to the strategy π, the expected value of rewards under the current strategy can be obtained as follows:(13)Vπst=EπRt+γVπSt+1,∀t<D−1.The Q value function is the action value function, which indicates the expected return after selecting the actionat:(14)Qπst,at=Rt+γVπst+1.Starting from the states0, the agent begins to search for the best strategy step by step, which maximizes the action value function:(15)π∗=argmaxπQπs,a,(16)Qπ=∗s,a=Qπ∗s,a.From equations (14)–(19), it can be seen that the model-free reinforcement learning method does not need to know the specific expression of the state transition probability function T but only needs to train according to the transition process of each interaction with the environment, so that the expected value of the total reward can reach the highest. The energy management and optimal dispatching method of power grid is transformed into Markov decision model. Then the rule-based dynamic energy management method is selected as the basic strategy, which can reduce the overall optimization cost, so as to avoid solving the optimal value function of all states. Then the adaptive learning control strategy is proposed, which can realize the continuous action control of energy management in intelligent park and the optimal dispatching decision in various scenarios.


## 3.2. Reinforcement Learning Framework for Energy Management in Intelligent Parks

In this paper, the intelligent park system is the agent’s environment. The agent optimizes the scheduling by adjusting the output and power generation of related aggregation units in the system. In timeT, the environment provides the observed system state st∈S to the agent, and the agent generates action DD based on strategy st∈S and intelligent park system state at∈A.


### 3.2.1. State Space

The state space is composed of the reference information of the decision-making of the agent at each moment, which mainly includes controllable state variableSC and external state variable SX. Among them, the controllable state variables are mainly environmental variables that can be directly/indirectly controlled by agents, such as the average SOC value of temperature-controlled load, the maximum SOC value of ESS, and the load electricity price level δt. External state variables are variables beyond the control of agents, such as temperature Tt and wind turbine power generation and higher-level market purchase and sale price Gt and Ptu,Ptd. Therefore, the state space can be described as(17)st∈S=SCxSXst=SoCt,BSCt,Gt,λtu,λtd,Lb,t,t.


### 3.2.2. Action Space

The action space of the agent is mainly composed of four parts, including temperature control load action spaceAtcl, load price action space AP, energy storage charge and discharge action space AD, and market purchase and sale electricity action space AE, which are specifically expressed as follows:(18)at∈A=AtclxApxADxAEat=Pt,ut+1i,Ct,Dt,Ptbuy,Ptsell.


### 3.2.3. Reward Function

The reward function is set to maximize the operational benefits of the intelligent park. The calculation method is to subtract the power generation cost from the income from selling electricity to the external power grid and the cost of purchasing electricity from the external power grid. The expression of the reward function is as follows:(19)Rt=St−Ct.St=Pt∑loadsLti+Cgen∑tclsLtcliuti+λtdPtsell.Costst=CgenGt+λtu+CtrimpPtbuy+CtrexpPtsell,where λtu,λtd are lower standby and upper standby electricity prices, which are the transaction electricity prices between microgrid and superior power grid. Gt is the power generation of wind turbines, Cgen is the power generation cost, and Ctrimp and Ctrexp are the energy transmission cost of purchasing electricity from or selling electricity to the power grid.


## 3.2.1. State Space

The state space is composed of the reference information of the decision-making of the agent at each moment, which mainly includes controllable state variableSC and external state variable SX. Among them, the controllable state variables are mainly environmental variables that can be directly/indirectly controlled by agents, such as the average SOC value of temperature-controlled load, the maximum SOC value of ESS, and the load electricity price level δt. External state variables are variables beyond the control of agents, such as temperature Tt and wind turbine power generation and higher-level market purchase and sale price Gt and Ptu,Ptd. Therefore, the state space can be described as(17)st∈S=SCxSXst=SoCt,BSCt,Gt,λtu,λtd,Lb,t,t.


## 3.2.2. Action Space

The action space of the agent is mainly composed of four parts, including temperature control load action spaceAtcl, load price action space AP, energy storage charge and discharge action space AD, and market purchase and sale electricity action space AE, which are specifically expressed as follows:(18)at∈A=AtclxApxADxAEat=Pt,ut+1i,Ct,Dt,Ptbuy,Ptsell.


## 3.2.3. Reward Function

The reward function is set to maximize the operational benefits of the intelligent park. The calculation method is to subtract the power generation cost from the income from selling electricity to the external power grid and the cost of purchasing electricity from the external power grid. The expression of the reward function is as follows:(19)Rt=St−Ct.St=Pt∑loadsLti+Cgen∑tclsLtcliuti+λtdPtsell.Costst=CgenGt+λtu+CtrimpPtbuy+CtrexpPtsell,where λtu,λtd are lower standby and upper standby electricity prices, which are the transaction electricity prices between microgrid and superior power grid. Gt is the power generation of wind turbines, Cgen is the power generation cost, and Ctrimp and Ctrexp are the energy transmission cost of purchasing electricity from or selling electricity to the power grid.


## 3.3. Energy Management Framework of Intelligent Park Based on DDPG Algorithm

The traditional reinforcement learning algorithm based onQ-learning performs well in dealing with small-scale discrete space problems. However, when dealing with practical problems with continuous state variables, with the increase of space dimensions, the discrete state variables will increase exponentially, which brings serious dimension disaster and cannot realize efficient learning of agents. In the energy management of intelligent park in this paper, firstly, the state variables such as wind turbine output, user load value, and market trading electricity are all continuous variables. In addition, the market transaction electricity quantity and energy storage charging and discharging power in the action space are also continuous variables, so the traditional reinforcement learning algorithm cannot effectively solve the energy management problem of intelligent park.Based on the above considerations, this paper uses deep neural network (DNN) to approximate reinforcement learning, so as to solve the energy management problem of intelligent park with continuous state/action space [33]. The DDPG algorithm based on the actor-critical framework is adopted in this paper. It uses two independent networks to approximate the critical function θQ and the actor function θQ, and each network has its own target networks θQ′ and θπ‘, for which the target Q value and target strategy are denoted as Q′ and π′.


### 3.3.1. Value Network Training

For value networks, the network parameters are usually optimized by minimizing loss function:(20)LθQ=Eyt−Qst,at|θQ2,where yt is the target Q value; the calculation method is as follows:(21)yt=rt+γQ′st+1,π′st+1|θπ′|θQ′.The value network parameters are updated in the gradient direction, andLθQ is calculated as follows about θQ gradient:(22)∇θQLθQ=E2yt−Qst,at|θQ∇θQQst,at.According to the gradient rule, the update formula of the value network parameters is as follows:(23)θQ←θQ−μQ∇θQLθQ.


### 3.3.2. Strategic Network Training

The training and updating of the strategy network are also based on gradient information. The gradient information of the strategy network is recorded as∇aQst,at|θQ, and the sampling strategy gradient is calculated as follows:(24)∇θππ=∇aQs,a|θQ|s=st,a=πst∇θππs,|θπ|s=st.According to the above deterministic policy gradient, the policy network parameters can be updated:(25)θπ←θπ−μπ∇θππ,where μπ is the strategic network learning rate.The target network parametersθQ′ and θπ′ sampling soft update technology further improve the stability of the learning process:(26)θQ′←τθQ−1−τθQ′.θπ′←τθπ−1−τθπ′,where τ is the soft renewal coefficient.


## 3.3.1. Value Network Training

For value networks, the network parameters are usually optimized by minimizing loss function:(20)LθQ=Eyt−Qst,at|θQ2,where yt is the target Q value; the calculation method is as follows:(21)yt=rt+γQ′st+1,π′st+1|θπ′|θQ′.The value network parameters are updated in the gradient direction, andLθQ is calculated as follows about θQ gradient:(22)∇θQLθQ=E2yt−Qst,at|θQ∇θQQst,at.According to the gradient rule, the update formula of the value network parameters is as follows:(23)θQ←θQ−μQ∇θQLθQ.


## 3.3.2. Strategic Network Training

The training and updating of the strategy network are also based on gradient information. The gradient information of the strategy network is recorded as∇aQst,at|θQ, and the sampling strategy gradient is calculated as follows:(24)∇θππ=∇aQs,a|θQ|s=st,a=πst∇θππs,|θπ|s=st.According to the above deterministic policy gradient, the policy network parameters can be updated:(25)θπ←θπ−μπ∇θππ,where μπ is the strategic network learning rate.The target network parametersθQ′ and θπ′ sampling soft update technology further improve the stability of the learning process:(26)θQ′←τθQ−1−τθQ′.θπ′←τθπ−1−τθπ′,where τ is the soft renewal coefficient.


## 3.4. Improved DDPG Algorithm

### 3.4.1. Improvement of Convergence Mechanism

(1) Random Exploration. In this paper, random noise is superimposed on the output actions to explore the environment; that is, in the process of training, the actions taken every time are improved as follows:(27)a^t=pat+1−pn,where a^t is the actual action after updating, and n is the noise that obeys the truncated normal distribution within the range of [−1, 1]; p is the proportion of neural network output action value at. The larger the value, the lower the randomness of exploration. In order to avoid any exploration at all, the upper limit of its value is 0.95.(2). Adjust the Proportion of Environmental Perception and Learning. In the traditional DDPG algorithm, every time an agent interacts with the environment, it needs to learn the model parameters once. In fact, this frequent learning time greatly increases the training time. Without sufficient interaction and exploration of the environment, frequent learning can easily make the agent fall into the local optimal value. In this paper, we adjust this and set the agent to learn every 30 times when it interacts with the environment.


### 3.4.2. Adding a Playback Pool for High Access Experience

Due to the high proportion of superimposed exploration in the output actions in the early stage of model training, there are few transfer strategies with higher total return value, and the model training speed is slow, and the influence of random exploration is even higher than that of scheduling sequence itself. In order to improve the convergence speed of the algorithm, an experience playback pool for storing high-quality transfer process is added, and its admission condition is set to be higher than the average value of the total reward value in each training cycle, so as to accelerate the convergence speed of the previous algorithm.In fact, the three measures basically do not change the complexity of the algorithm from the mathematical model, only by adjusting the relevant parameters, or some of the results have been trained to optimize, because the mathematical model does not increase the complexity of the algorithm.In summary, the basic framework of solving energy management problems in intelligent parks based on improved DDPG algorithm is shown in Figure4. Under this framework, the input is state variables, including market electricity price information, wind turbine output, and other state variables, and the output is action vectors, which mainly include action variables such as charging and discharging power of electric energy storage system, power consumed by TCL, demand response load, and market purchase and sales power.Figure 4
Structure diagram of community microgrid optimal scheduling model based on improved A3C algorithm.(3). Example Test and Result Analysis. In this paper, a toolkit named Gym of Open AI is used to build the simulation environment of intelligent park. TensorFlow 1.1.4 Toolkit is used to train neural network, and the training data of wind power and market electricity price are taken from [34].In terms of model parameter setting, the charging and discharging efficiency of ESS is 0.9, the maximum charging and discharging power is set to 250 kW, the capacity is set to 500kWh, the wind power generation cost is 32/MW, the upper standby price and the lower standby price are taken from [35], Ctrimp and Ctrexp are 0.9/MW and 1.3/MW, respectively, and TCL parameters are taken from [36]. The total quantity is 100, the total period of a day is 24 h, and the retail electricity price is 5.48 euro/kW. All experimental results are average by multiple times.The structure of DDPG network is shown in Appendix in Figures5 and 6. Considering the dimensions of state space and action space, the number of hidden layer neurons of actor network is 300/100/100, and the number of hidden layer S1, A1, H1, H2 neurons of critic network is 200, 50, 100, and 100, respectively. ReLU function is used as the activation function of hidden layer. The learning rate of actor network and critic network is 0.001 and 0.0001, respectively, the learning rate of target network is 0.001, and the capacity of experience pool is 105.Figure 5
Structure of critic network and its target network.Figure 6
Structure of actor network and its target network.


## 3.4.1. Improvement of Convergence Mechanism

(1) Random Exploration. In this paper, random noise is superimposed on the output actions to explore the environment; that is, in the process of training, the actions taken every time are improved as follows:(27)a^t=pat+1−pn,where a^t is the actual action after updating, and n is the noise that obeys the truncated normal distribution within the range of [−1, 1]; p is the proportion of neural network output action value at. The larger the value, the lower the randomness of exploration. In order to avoid any exploration at all, the upper limit of its value is 0.95.(2). Adjust the Proportion of Environmental Perception and Learning. In the traditional DDPG algorithm, every time an agent interacts with the environment, it needs to learn the model parameters once. In fact, this frequent learning time greatly increases the training time. Without sufficient interaction and exploration of the environment, frequent learning can easily make the agent fall into the local optimal value. In this paper, we adjust this and set the agent to learn every 30 times when it interacts with the environment.


## 3.4.2. Adding a Playback Pool for High Access Experience

Due to the high proportion of superimposed exploration in the output actions in the early stage of model training, there are few transfer strategies with higher total return value, and the model training speed is slow, and the influence of random exploration is even higher than that of scheduling sequence itself. In order to improve the convergence speed of the algorithm, an experience playback pool for storing high-quality transfer process is added, and its admission condition is set to be higher than the average value of the total reward value in each training cycle, so as to accelerate the convergence speed of the previous algorithm.In fact, the three measures basically do not change the complexity of the algorithm from the mathematical model, only by adjusting the relevant parameters, or some of the results have been trained to optimize, because the mathematical model does not increase the complexity of the algorithm.In summary, the basic framework of solving energy management problems in intelligent parks based on improved DDPG algorithm is shown in Figure4. Under this framework, the input is state variables, including market electricity price information, wind turbine output, and other state variables, and the output is action vectors, which mainly include action variables such as charging and discharging power of electric energy storage system, power consumed by TCL, demand response load, and market purchase and sales power.Figure 4
Structure diagram of community microgrid optimal scheduling model based on improved A3C algorithm.(3). Example Test and Result Analysis. In this paper, a toolkit named Gym of Open AI is used to build the simulation environment of intelligent park. TensorFlow 1.1.4 Toolkit is used to train neural network, and the training data of wind power and market electricity price are taken from [34].In terms of model parameter setting, the charging and discharging efficiency of ESS is 0.9, the maximum charging and discharging power is set to 250 kW, the capacity is set to 500kWh, the wind power generation cost is 32/MW, the upper standby price and the lower standby price are taken from [35], Ctrimp and Ctrexp are 0.9/MW and 1.3/MW, respectively, and TCL parameters are taken from [36]. The total quantity is 100, the total period of a day is 24 h, and the retail electricity price is 5.48 euro/kW. All experimental results are average by multiple times.The structure of DDPG network is shown in Appendix in Figures5 and 6. Considering the dimensions of state space and action space, the number of hidden layer neurons of actor network is 300/100/100, and the number of hidden layer S1, A1, H1, H2 neurons of critic network is 200, 50, 100, and 100, respectively. ReLU function is used as the activation function of hidden layer. The learning rate of actor network and critic network is 0.001 and 0.0001, respectively, the learning rate of target network is 0.001, and the capacity of experience pool is 105.Figure 5
Structure of critic network and its target network.Figure 6
Structure of actor network and its target network.


## 3.5. Comparison of Convergence Speed and Learning Effect of Different Algorithms

In order to show the effectiveness of the improved DDPG algorithm proposed in this paper, the following basic comparative examples are set, as shown in Table1.Table 1
Final reward of each agent in different cases.
OptionRandomPerceptual learning ratioExperience poolsReward valueTraining time (h)1×110.16221.62√110.19419.543√30 : 110.22613.654√30 : 120.2312.16As can be seen from the table, compared with scheme 1 and scheme 2, it can be seen that the reward value is increased by about 19.75% and the training time is reduced by nearly 2 hours after adopting the random exploration strategy proposed in this paper. In addition, comparing scheme 3 and scheme 2, set the ratio of environment awareness and agent learning to 30: After 1, not only has the reward value of the agent been greatly increased, but also the training time has been obviously reduced, which is mainly due to the improvement of environment perception and agent learning ratio, and the reduction of a large number of unnecessary learning times, so the convergence speed of the algorithm is significantly improved, and the agent is prevented from falling into local optimal solution. Finally, compared with scheme 4 and scheme 3, it can be seen that, after adding experience pool, the algorithm further improves the convergence speed and reduces the training time on the premise of ensuring a high reward value. The reward curves under scenarios 1, 2, 3, and 4 are shown in Figure7.Figure 7
Reward curves in training process of different algorithms.Figure7 shows the reward curves of the four different schemes in Table 1, with different scheme configurations resulting in different curve trends. Under the random exploration strategy, further refine the final reward value of agents under different perceptual learning ratios, as shown in Figure 8. As can be seen from the figure, although the perceptual learning ratio can greatly reduce the training time of the model, avoid the model falling into local optimum, but when the training is carried out to a certain extent, too high proportion of perceptual exploration will lose more key information. As a result, the overall reward value is reduced and the training time is increased. Under the four comparison schemes, when the perceptual learning ratio is 30 : 1, the training time and the final reward value are optimal. Therefore, in the process of adjusting the parameters of the time model, it is necessary to set the perceptual learning ratio more in line with the model training in combination with practical problems.Figure 8
Relationship between training time and final reward value.In order to demonstrate the effectiveness of the improved DDPG algorithm compared with the existing one, the experimental results of the existing algorithms compared with the DQN algorithm, the actor-critic algorithm, and the PPO algorithm under different hyperparameters are tested respectively, as shown in Table2.Table 2
Experimental results of each algorithm under different super parameters.
Greedy coefficient (E)εDQNActor-criticPPODDPG1.0–40.0910.052−0.1520.1915.0–50.1230.075−0.1220.2451.0–50.1340.062−0.0910.2175.0–60.1390.109−0.0740.203As can be seen from the table, under different greedy coefficients, the final reward values of each algorithm are also different, showing different trends. The PPO algorithm has the worst adaptability to the model, and the overall learning effect is the worst. The DQN algorithm is better than actor-critic algorithm, but it lags behind the proposed algorithm. In addition, with the decreasing of the greedy coefficient, the learning effect of each algorithm is improved. Actor-critic algorithm and DDPG algorithm need the appropriate greedy coefficient to achieve the optimal learning effect.From Table3, we can see that the algorithms exhibit different performance with the change of exploring coefficient in the strategy of random exploration. On the whole, each algorithm can use the better learning effect between 0.5 and 0.75; the higher the exploring coefficient, the lower the randomness of the searching algorithm, and it is easy to fall into the local convergence, leading to insufficient learning, poor learning effect, and low exploring coefficient, which will lead to too random and frequent exploration, not only seriously affecting the convergence time of the algorithm, but also leading to insufficient key information learning and poor learning effect.Table 3
Experimental results of each algorithm under different super parameters.
Exploring coefficientPDQNActor-criticPPODDPG0.05—0.043−0.9220.1530.25—−0.024−0.0890.1970.5—0.075−0.9220.2450.75—0.062−0.9220.2130.95—0.014−0.9580.128In terms of training time and complexity of the algorithm, when setting the same hyperparameter, the comparison results of the four algorithms are shown in Table4.Table 4
Experimental results of each algorithm under different super parameters.
AttributeDQNActor-criticPPODDPGReward value0.1340.062−0.0910.217Training time (h)20.7218.1915.3712.16From the point of view of training complexity, the improved DDPG algorithm has a fast training speed due to the double-sample pool and reasonable environment perception-learning ratio and can keep good learning effect under the fast training speed.


## 3.6. Comparison of Test Results

Fix the trained neural network weights, and select the actual data of a typical day in winter for testing. See Figures9–11 for the temperature, basic load, market electricity price information, and wind power output value of that day, and set the following comparative examples.Figure 9
Load curve and temperature curve.Figure 10
Wind power output result.Figure 11
Electricity purchase price and electricity sale price curve.Case 1:.
Genetic algorithm is used to test the typical daily data; Case 2: The traditional DDPG algorithm is used to test the typical daily data (the algorithm corresponds to scheme 1 in Section3.1); Case 3: The improved DDPG algorithm proposed in this paper is used to test the typical day (algorithm comparison, Section 3.1, scheme 4).
In the above examples, the result obtained by Case 1 can be considered as the optimal control result. The total economic benefits of intelligent parks and the calculation time of algorithms under different examples are shown in Table5.
As can be seen from Table5, under three examples, the total income of the park obtained by genetic algorithm is the highest, but at the same time, its calculation time is also the longest. The final benefit values of the two schemes using deep reinforcement learning algorithm are quite different. After training based on the traditional DDPG algorithm, the final test effect is poor, and the total income value is only 339.19, far lower than the genetic algorithm and the improved DDPG algorithm proposed in this paper. In addition, it is worth noting that deep reinforcement learning is essentially a model-free algorithm. It can learn decision-making experience from massive historical data and continuously improve the decision-making ability of agents, although the final test results are not as good as the traditional model-based optimization algorithm. However, the improved DDPG algorithm can greatly reduce the calculation time and even realize the decision-making ability of millisecond response, which cannot be realized by the traditional model-based solution algorithm, on the premise that the calculation results are not much different from the optimal control algorithm. In order to prove this conclusion, under the same test environment, the scale of solving the problem is further expanded. The comparison of computing time of three algorithms and the other DRL methods is shown as Table 6.
It can be seen that, with the continuous expansion of the scheduling cycle, the difference in solution time of the three algorithms is more obvious. Because of the offline learning and online decision-making ability of deep reinforcement learning, the calculation time basically increases linearly with the scheduling cycle, while the calculation time of traditional intelligent algorithms increases exponentially with the expansion of the scheduling cycle. With the increasing number of distributed new energy and controllable load connected to the intelligent park in the future, the overall dispatching scale of the park will gradually expand in the future, and the advantages of deep reinforcement learning algorithm will be further reflected at this time.
In addition, by comparing the test results of different algorithms, under different scheduling cycles, the target function values are shown respectively in Table7.
As can be seen from the table, the test results of each algorithm are basically the same as the training results. The improved DDPG algorithm has the highest profit value in all the test algorithms because of its good learning effect in the training stage, and the PPO algorithm also has the lowest profit value in the test stage because of its poor learning effect.
Figure12 shows the optimization results of interactive power consumption between intelligent park and superior power grid under different examples. It can be seen from the figure that, under the problem framework of this paper, EMS system chooses to sell electricity to the superior power grid in 4–10 h when the market electricity price is higher and purchases electricity from the superior power grid in 16–22 h when the market electricity price is lower, which not only meets its aggregate load resource electricity demand, but also maximizes its operating income. In addition, although both the traditional DDPG algorithm and the improved DDPG algorithm can keep the same change level with the optimal results, the improved DDPG algorithm is obviously closer to the optimal results in numerical results, and the control effect is better than the basic DDPG algorithm.
In addition, in order to reflect the impact of load side regulation on the overall income of the park, the following comparative examples are set up, respectively:Case 1: consider regulating price-based demand response resources and temperature-controlled load resources simultaneouslyCase 2: only consider regulating price-based demand response resourcesCase 3: only consider regulating temperature-controlled load resourcesCase 4: do not consider any load side resources
Under four scenarios, the results of revenue and total transactions with the grid under the 24-hour scheduling cycle are shown in Table8.
From the results, it can be concluded that considering user-side resources in the form of demand response can effectively reduce the operation cost of the park. By responding positively in the period of higher electricity price and buying a large amount of electricity in the period of low electricity price to store energy for temperature-controlled load, the load curve of the park can be improved greatly while improving the economic benefit of the park. The specific equivalent load curve is shown in Figure13.Table 5
Total benefit of the park and the calculation time of the algorithm in different cases.
ParameterCase 1Case 2Case 3Profit (€)428.35339.19413.26Computing time (s)5.890.0650.039Table 6
Calculation time of three algorithms under different scheduling period.
Scheduling period (h)Case 1Case 2Case 3DQNPPO245.890.0650.0390.0410.038168216.560.5360.1560.1760.1647201967.252.681.5641.4791.625Table 7
Profit result of three algorithms under different scheduling period.
Scheduling period (h)DQNActor-criticPPODDPG24358.24319.27148.87413.261682301.872008.791374.432668.8272010820.179776.327876.9112208.32Figure 12
The results of market trading electricity under different algorithms.Table 8
Test result of three algorithms under different scheduling period.
Scheduling periodCase 1Case 2Case 3Case 4Profit (€)413.26379.27356.99298.76Trading electricity (kW)2601.892508.661887.381765.62Figure 13
Equivalent load curve under different examples.


## 4. Conclusion

In this paper, an energy management model of intelligent park considering the price-based demand response load and temperature-controlled load on the user side is established, and a model-free deep reinforcement learning algorithm is used to solve the problem. After analysis and verification, the following conclusions are drawn:(1)
Based on the traditional DDPG algorithm, the convergence speed of the algorithm and the final reward value can be improved by random exploration strategy, changing the perceptual learning ratio and increasing the high admission experience pool to a certain extent.(2)
Appropriate perceptual learning ratio can not only greatly reduce the training time of the model, but also avoid the algorithm falling into local optimal solution. However, the setting of perceptual learning ratio should be moderate, and too large or too small perceptual learning ratio is not conducive to the improvement of training effect.(3)
The model-free reinforcement learning algorithm can realize offline learning and online application. After the model is trained and converged, it can be applied to the online scheduling decision of intelligent park, and the calculation time is much less than that of the model-based solution algorithm.



---

*Source: 6177978-2022-02-14.xml*