# Neural Network-Based Intelligent Computing Algorithms for Discrete-Time Optimal Control with the Application to a Cyberphysical Power System

Authors: Feng Jiang; Kai Zhang; Jinjing Hu; Shunjiang Wang
Journal: Complexity
 (2021)

---

## Abstract

Adaptive dynamic programming (ADP), which belongs to the field of computational intelligence, is a powerful tool to address optimal control problems. To overcome the bottleneck of solving Hamilton–Jacobi–Bellman equations, several state-of-the-art ADP approaches are reviewed in this paper. First, two model-based offline iterative ADP methods including policy iteration (PI) and value iteration (VI) are given, and their respective advantages and shortcomings are discussed in detail. Second, the multistep heuristic dynamic programming (HDP) method is introduced, which avoids the requirement of initial admissible control and achieves fast convergence. This method successfully utilizes the advantages of PI and VI and overcomes their drawbacks at the same time. Finally, the discrete-time optimal control strategy is tested on a power system.


---

## Body

## 1. Introduction

Adaptive dynamic programming (ADP) [1–4], which integrates the advantages of reinforcement learning (RL) [5–8] and adaptive control, has become a powerful tool in solving optimal control problems. With decades of development, ADP has also provided many approaches to solve other control problems, such as robust control [9, 10], optimal control with input constraints [11, 12], optimal tracking control [13, 14], zero-sum games [15], and non-zero-sum games [16]. Furthermore, ADP methods have been widely applied to the real-world systems, such as water-gas shift reaction [17], battery management [18], microgrid systems [19, 20], and Quanser helicopter [21]. These aforementioned papers were all inspired and developed by the basic works of ADP-based optimal control; i.e., optimal control is the core research topic of ADP.The bottleneck of solving the nonlinear optimal control problems is to obtain the solutions of Hamilton–Jacobi–Bellman (HJB) equations. However, these equations are generally difficult or even impossible to be solved analytically. To overcome this difficulty, ADP has given several important iterative learning frameworks, such as policy iteration (PI) [2, 22, 23] and value iteration (VI) [24–26]. PI algorithm starts from an initial admissible control policy and then proceeds the policy evaluation step and the policy improvement step successively till convergence. The main advantage of PI is that it ensures all the iterative control policies are admissible and achieves fast convergence. The drawback of PI is also obvious. The requirement of initial admissible control is a strict condition in practice, which seriously limits its applications. Different from PI, VI can start from an arbitrary-positive semidefinite value function, which is an easy-to-realize initial condition. Although the easier initial condition makes VI more practical, it also leads to a longer iteration learning process; that is, VI achieves convergence much slower than PI. Thus, it is desired to develop a new method, which avoids the requirement of initial admissible control and gets convergence faster than the VI algorithm. To realize these purposes, the multistep heuristic dynamic programming (HDP) approach [27] is presented to integrate the merits of PI and VI algorithms and overcome their drawbacks.This paper reviews the state-of-the-art ADP algorithms for the optimal control of discrete-time (DT) systems. The rest of this paper is arranged as follows. In Section2, the problem formulation is derived. Three iterative model-based offline learning algorithms along with comprehensive comparisons are presented in Sections 3 and 4. The proposed DT optimal control strategy is tested on a power system in Section 5. Finally, a brief conclusion is drawn in Section 6.


## 2. Problem Formulation

In this paper, we consider the general nonlinear DT system:(1)xk+1=fxk+gxkuk,where xk∈ℝn represents the system state, uk∈ℝm denotes the control input, and fx∈ℝn and gx∈ℝn×m are the system functions.The purpose of the optimal control issue is to find out a state feedback control policyuxk, which can not only stabilize system (1) but also minimize the following performance index function:(2)Jx0,u=∑p=0∞rxp,up,where rx,u=xTQx+uTRu. The matrices Q and R determine the performance of system states and control inputs, respectively. Given the admissible control policy uxk, the value function can be described by(3)Vxk=∑p=k∞rxp,up=rxk,uk+∑p=k+1∞rxp,up=rxk,uk+Vxk+1.According to the definition of optimal control, the optimal value function can be defined by(4)V∗xk=minux∑p=k∞rxp,up=minuxrxk,uk+V∗xk+1.By using the stationarity condition [28], the optimal control policy can be derived as(5)u∗xk=argminuxrxk,uk+V∗xk+1=argminuxrxk,uk+V∗fxk+gxkuk=−12R−1gTxk∇V∗xk+1,where ∇V∗xk+1=∂V∗xk+1/∂xk+1.The key to obtaining the optimal control policyu∗xk is to solve the following DT HJB equation [27]:(6)V∗xk=rxk,u∗k+V∗xk+1=14∇V∗Txk+1gxkR−1gTxk∇V∗xk+1+xTkQxk+V∗xk+1.Remark 1.
Figure1 provides the relationship and difference between discrete-time and continuous-time optimal control. The real-world systems generally exist in the continuous-time forms. After mathematical modeling, they are formulated by the continuous-time system models. Through sampling and discretization, the continuous-time system models are converted into the discrete-time ones. Therefore, the associated performance indexes and HJB equations of discrete-time systems are in the discretization forms compared with the continuous-time systems. The key to solving the discrete-time optimal control issue is the discrete-time HJB equation, which is a nonlinear partial difference equation. The existing works regarding continuous-time systems are much more than the ones regarding discrete-time systems. In order to overcome this bottleneck, several ADP learning algorithms along with their neural network (NN) implementations will be introduced.Figure 1
Relationship and difference between discrete-time and continuous-time optimal control.


## 3. Model-Based PI Algorithm for the Optimal Control Problem of DT Systems

In this section, the model-based PI algorithm along with its NN implementation will be introduced in detail. The model-based PI algorithm [2, 23] is shown in Algorithm 1.Algorithm 1: Model-based PI.
Step 1: (Initialization)Let the iteration indexl=0.Select an initial admissible control policyu0x.Choose a small enough computation precisionϵ.Step 2: (Policy Evaluation)Withulx, compute the iterative value function Vlx byVlxk=rxk,ulk+Vlxk+1.Step 3: (Policy Improvement)WithVlx, update the iterative control policy ul+1x byul+1xk=−1/2R−1gTxk∇Vlxk+1.Step 4: if Vl−Vl−1≤ɛ, stop and the optimal control policy ul+1x is acquired;Else, letl=l+1 and go back to Step 2.The actor-critic dual-network structure with the gradient-descent updating law is employed to implement Algorithm1. First, construct the critic NN to approximate the iterative value function:(7)V^l,qxk=W^cl,qTϕcxk,where W^cl,q and ϕcx denote the NN weights and NN activation functions of the critic network and q is the iteration index for the following gradient-descent method.Define the error function for the critic NN:(8)V˜l,qxk=V^l,qxk−Vlxk,where Vlxk=∑h=0H−1rxk+h,ulk+h+Vlxk+H. If we select a large enough integer H, then, with the admissible control ulk, one has Vlxk+H⟶0 [2]; that is, Vlxk can be expressed as Vlxk=∑h=0H−1rxk+h,ulk+h.In order to minimize the error performanceEcl,qk=V˜l,qxk2/2, the gradient-descent-based updating law for the critic NN is given by(9)W^cl,q+1=W^cl,q−βc∂Ecl,qk∂V˜l,qxk∂V˜l,qxk∂V^l,qxk∂V^l,qxk∂W^cl,q=W^cl,q−βcV˜l,qxkϕcxk,where βc is the learning rate of the critic NN.Similar to the design of critic NN, the actor network, which is used to approximate the iterative control policy, is expressed as(10)u^l,qxk=W^al,qTϕaxk.The error function for the actor NN is defined as(11)u˜l,qxk=u^l,qxk−ulxk,where ulxk can be attained according to Algorithm 1.To minimize the error performanceEal,qk=u˜l,qTxku˜l,qxk/2, using the chain derivation rule, the updating law for the actor NN is designed by(12)W^al,q+1=W^al,q−βa∂Eal,qk∂u˜l,qxk∂u˜l,qxk∂u^l,qxk∂u^l,qxk∂W^al,q=W^al,q−βaϕaxku˜l,qTxk,where βa is the learning rate of the actor NN.Remark 2.
Figure2 displays the NN implementation diagram of PI algorithm. First, NN weights of the actor network should be chosen to generate admissible control. Second, critic and actor networks are updated via the gradient-descent-based learning law to realize policy evaluation and improvement steps, respectively. After iteration, critic and actor networks achieve convergence, where the NN-based approximate optimal control can be obtained. Many stability proofs of the NN implementation procedure have been given in the existing works. Here, we introduce the following rigorous proof to demonstrate the optimality and convergence.Figure 2
NN implementation diagram of PI algorithm.Theorem 1.
Let the target iterative value function and control policy be described byVlxk=WclTϕcxk and ulxk=WalTϕaxk, respectively. Let the critic and actor NNs be updated via (9) and (12), respectively. If the learning rates βc and βa are selected to be appropriately small, then the NN weights W^cl,q and W^al,q will asymptotically converge to the ideal values Wcl and Wal, respectively.Proof.
LetW˜cl,q=W^cl,q−Wcl and W˜al,q=W^al,q−Wal. According to (9) and (12), it can be acquired that(13)W˜cl,q+1=W˜cl,q−βcV˜l,qxkϕcxk,W˜al,q+1=W˜al,q−βaϕaxku˜l,qTxk,where V˜l,qxk=W˜cl,qTϕcxk and u˜l,qxk=W˜al,qTϕaxk.
Construct the following Lyapunov function candidate:(14)PW˜cl,q,W˜al,q=trW˜cl,qTW˜cl,q+W˜al,qTW˜al,q.
The difference of the Lyapunov function (14) can be derived as(15)ΔPW˜cl,q,W˜al,q=trW˜cl,q+1TW˜cl,q+1−W˜cl,qTW˜cl,q+trW˜al,q+1TW˜al,q+1−W˜al,qTW˜al,q≤βcV˜l,qxk2βcϕcxk2−2+βau˜l,qxk2βaϕaxk2−2.
If the learning rates are selected to satisfyβc≤2/ϕcxk2 and βa≤2/ϕaxk2, then one has ΔPW˜cl,q,W˜al,q≤0, which implies the NN weights W^cl,q and W^al,q will asymptotically converge to the ideal values.
This completes the proof.


## 4. Model-Based VI Algorithm and Multistep HDP Algorithm

With the help of the initial admissible control, the PI algorithm achieves fast convergence. However, the weakness of the PI algorithm is obvious. The PI algorithm requires the initial control policy to be admissible, which is a strict condition. How to find out an initial admissible control policy is still an open problem, which limits the real-world applications of the PI algorithm. To relax the strict condition, the model-based VI algorithm [24–26] is shown in Algorithm 2, where the initial condition becomes much easier.Algorithm 2: Model-based VI.
Step 1: (Initialization)Let the iteration indexl=0.Select an initial value functionV0x.Choose a small enough computation precisionϵ.Step 2: (Policy Improvement)WithVlx, compute the iterative control policy ulx byulxk=−1/2R−1gTxk∇Vlxk+1.Step 3: (Policy Evaluation)Withulx, calculate the iterative value function Vl+1x byVl+1xk=rxk,ulk+Vlxk+1.Step 4: if Vl+1−Vl≤ɛ, stop and the optimal control policy ulx is acquired;Else, letl=l+1 and go back to Step 2.Remark 3.
Different from the PI algorithm, the VI algorithm does not require the initial admissible control, and one only needs to provide a specific initial value function, which makes the VI algorithm more practical in the real-world applications. However, without the help of the initial admissible control, the VI algorithm generally suffers from the low convergence speed. From the aforementioned content, it can be observed that the PI and VI algorithms have their own advantages and disadvantages. The PI algorithm can achieve fast convergence, while it requires an initial admissible control policy. The VI algorithm can start from an easy-to-realize initial condition, while it generally suffers from the low convergence speed. Thus, it is expected to design a new approach, which can make the trade-off between the PI algorithm and the VI algorithm.
That is, it is desired to develop an algorithm, which achieves convergence faster than the VI algorithm and does not require an initial admissible control policy. To realize this goal, the multistep HDP method [27] will be introduced in Algorithm 3.
Construct the critic and actor NNs to approximate the iterative value function and control policy as follows:(16)V^lxk=ϕcTxkW^cl,u^lxk=ϕaTxkW^al,where W^cl and W^al are the NN weights and ϕcx and ϕax are the associated NN activation functions.
According to Algorithm3, using the NNs to estimate the solutions will yield the following error:(17)ecl=ϕcTxkW^cl+1−∑p=kk+α−1rxp,u^lp−ϕcTxk+αW^cl.
Letθ=ϕcTxk and σl=∑p=kk+α−1rxp,u^lp+ϕcTxk+αW^cl. Equation (17) becomes(18)ecl=θW^cl+1−σl.
To minimizeecl2, we employ the least-square method to update W^cl+1. Collect M different data sets for training, where M is a large enough number. Then, one has Π=θ1T,θ2T,…,θMTT and Θl=σ1lT,σ2lT,…,σMlTT. The least-square-based updating law for W^cl+1 is given by(19)W^cl+1=ΠTΠ−1ΠTΘl.
To minimizerxk,u^lxk+V^lxk+1, the gradient-descent-based updating law for the actor NN is given by(20)W^al,q+1=W^al,q−βa∂rxk,u^l,qxk+V^lxk+1∂u^l,qxk∂u^l,qxk∂W^al,q=W^al,q−βaϕaxk2Ru^l,qxk+gTxk∂ϕcxk+1∂xk+1W^cl.Algorithm 3: Multistep HDP.
Let the iteration indexl=0.Select an initial value functionV0x.Choose a small enough computation precisionϵ.Step 2: (Policy Improvement)WithVlx, compute the iterative control policy ulx byulxk=−1/2R−1gTxk∇Vlxk+1.Step 3: (Multistep Policy Evaluation)Withulx, calculate the iterative value function Vl+1x byVl+1xk=∑p=kk+α−1rxp,ulp+Vlxk+α.Step 4: if Vl+1−Vl≤ɛ, stop and the optimal control policy ulx is acquired;Else, letl=l+1 and go back to Step 2.Remark 4.
From Table1 and Figure 3, we can see the performance comparison and relationship among PI algorithm, VI algorithm, and multistep HDP. Due to the existence of initial admissible control, the PI algorithm gets fast convergence. However, the condition of initial admissible control is difficult to realize. Different from the PI algorithm, the initial condition of VI algorithm is easy-to-realize. However, the initial condition may not be admissible, which may lead to the low stability. Multistep HDP follows the initial condition of VI algorithm and develops the multistep policy evaluation step to obtain more history data. Therefore, multistep HDP is easy-to-realize and achieves fast convergence at the same time; that is, multistep HDP successfully combines the advantages of PI and VI algorithms.Table 1
Performance comparison among PI algorithm, VI algorithm, and multistep HDP.
Methods and performanceComputation stabilityInitial difficultyConvergence speedPI algorithmHighDifficultHighVI algorithmLowEasyLowMultistep HDPMediumEasyMediumFigure 3
Relationship and difference among PI algorithm, VI algorithm, and multistep HDP.


## 5. Application to a Benchmark Power System

The benchmark power system investigated in this paper is illustrated in Figure4. This power system can be regarded as a microgrid, which is composed of nonpolluting energy (subsystems I and II), load demand sides (subsystem III), and regular generations (subsystem IV). The core control unit is the management center, which maintains the frequency stability against load variations.Figure 4
The benchmark power system considered in this paper.


### 5.1. System Model and Application

In Figure5, first, the real-world power system can be formulated by a state space function via mathematical modeling. After sampling and discretization, the system model can be controlled by computers. Through iterative ADP learning, the approximate optimal control can be obtained. Substituting the approximate optimal control into the system model will yield simulation results. To test the effectiveness of the proposed DT optimal control strategy, let us consider the following power system [19, 20]:(21)Δξf.=−1TpΔξf+αpTpΔξt,Δξt.=−1TtΔξt+1TtΔξg,Δξg.=−1αsTgΔξf−1TgΔξg+1Tgu,where Δξf is the frequency deviation; Δξt denotes the turbine power; Δξg represents the governor position value; Tt, Tg, and Tp denote the time constants of turbine, governor, and power system, respectively; αp represents the gain of power system; αs is the speed regulation coefficient; u denotes the control input; and x is the state variable. Let x=Δψf,Δψt,ΔψgT, where x1=Δψf,x2=Δψt, and x3=Δψg. Then, the system (21) can be discretized as the form of (1). Set the matrices in the performance index function: Q=2I3 and R=1.Figure 5
Application to the benchmark power system.


### 5.2. Simulation Results

Simulation results are shown in Figure6. Figure 6(a) implies the system states cannot be stabilized without control. Then, we apply the optimal control strategy into the system. Figure 6(b) indicates the system states can be stabilized after 8 time steps under optimal control. Comparing the trajectories of the system states, the superior control performance of optimal control strategy can be observed. Figure 6(c) shows the 2D plot of convergence trajectory in detail. Figure 6(d) provides the evolution of the control input. The aforementioned simulation results demonstrate the high stability, fast convergence, and low control cost of the DT optimal control strategy.Figure 6
Simulation results: (a) system states without control; (b) system states with optimal control; (c) 2D plot ofx1 and x2; (d) control input.
(a)(b)(c)(d)


## 5.1. System Model and Application

In Figure5, first, the real-world power system can be formulated by a state space function via mathematical modeling. After sampling and discretization, the system model can be controlled by computers. Through iterative ADP learning, the approximate optimal control can be obtained. Substituting the approximate optimal control into the system model will yield simulation results. To test the effectiveness of the proposed DT optimal control strategy, let us consider the following power system [19, 20]:(21)Δξf.=−1TpΔξf+αpTpΔξt,Δξt.=−1TtΔξt+1TtΔξg,Δξg.=−1αsTgΔξf−1TgΔξg+1Tgu,where Δξf is the frequency deviation; Δξt denotes the turbine power; Δξg represents the governor position value; Tt, Tg, and Tp denote the time constants of turbine, governor, and power system, respectively; αp represents the gain of power system; αs is the speed regulation coefficient; u denotes the control input; and x is the state variable. Let x=Δψf,Δψt,ΔψgT, where x1=Δψf,x2=Δψt, and x3=Δψg. Then, the system (21) can be discretized as the form of (1). Set the matrices in the performance index function: Q=2I3 and R=1.Figure 5
Application to the benchmark power system.


## 5.2. Simulation Results

Simulation results are shown in Figure6. Figure 6(a) implies the system states cannot be stabilized without control. Then, we apply the optimal control strategy into the system. Figure 6(b) indicates the system states can be stabilized after 8 time steps under optimal control. Comparing the trajectories of the system states, the superior control performance of optimal control strategy can be observed. Figure 6(c) shows the 2D plot of convergence trajectory in detail. Figure 6(d) provides the evolution of the control input. The aforementioned simulation results demonstrate the high stability, fast convergence, and low control cost of the DT optimal control strategy.Figure 6
Simulation results: (a) system states without control; (b) system states with optimal control; (c) 2D plot ofx1 and x2; (d) control input.
(a)(b)(c)(d)


## 6. Conclusions

In this paper, several state-of-the-art ADP-based methods have been reviewed to address the optimal control problem of DT systems. A comprehensive comparison has been made between PI and VI. A novel multistep HDP method has been introduced to integrate the advantages of PI and VI algorithms with either strict requirement of initial admissible control or longer interaction learning process. The simulation results have demonstrated the effectiveness of our proposed schemes.



---

*Source: 5549678-2021-04-17.xml*