# A Brief Overview of Optimal Robust Control Strategies for a Benchmark Power System with Different Cyberphysical Attacks

Authors: Bo Hu; Hao Wang; Yan Zhao; Hang Zhou; Mingkun Jiang; Mofan Wei
Journal: Complexity
 (2021)

---

## Abstract

Security issue against different attacks is the core topic of cyberphysical systems (CPSs). In this paper, optimal control theory, reinforcement learning (RL), and neural networks (NNs) are integrated to provide a brief overview of optimal robust control strategies for a benchmark power system. First, the benchmark power system models with actuator and sensor attacks are considered. Second, we investigate the optimal control issue for the nominal system and review the state-of-the-art RL methods along with the NN implementation. Third, we propose several robust control strategies for different types of cyberphysical attacks via the optimal control design, and stability proofs are derived through Lyapunov theory. Furthermore, the stability analysis with the NN approximation error, which is rarely discussed in the previous works, is studied in this paper. Finally, two different simulation examples demonstrate the effectiveness of our proposed methods.


---

## Body

## 1. Introduction

With the development of cloud computing, artificial intelligence, and 5th-generation, the power systems regarded as the primary infrastructures in society become typical CPSs [1, 2]. Since there are numerous physical sensors, complex interaction mechanisms, and massive signals in cyberphysical power systems [3], the security of CPSs is inevitably threatened. For example, a large-scale blackout caused by cyberattacks in Ukraine had disrupted the normal lives of many people [4, 5]. Despite there are many advanced control strategies in CPSs, the imperfection of security has not been sufficiently addressed. Meanwhile, the power system composed of distributed energy and multiple loads is multidimensional [6]. It is urgent to further strengthen the security of CPSs.Generally, the security of CPSs is threatened by attacks from the perception layer, cyberlayer, and decision layer. In particular, the attacks at the perception layer and cyberlayer, known as cyberattacks, severely disrupt the system. In recent years, reliable control strategies against various cyberattacks, such as false data injection attacks, time-delay switch attacks, and denial-of-service attacks, have been presented by many scholars. Denial-of-service attacks, which can jam information transmission channel, are an aggressive threat to CPS security [7–9]. A novel control strategy based on the game theoretic approach was proposed to resist the attacks in discrete systems [7]. Similar to the theory of literature [7], Seo et al. [9] primarily solved jamming attack in the communication between sensor and network, where an adaptive scheduling with energy constraints was presented. Using an evaluation function that quantitatively analyzes the impact of attacks, the optimal attack strategy was investigated under energy constraint in a wireless network, which can maximally destroy the stability of the system [10]. Besides, false data injection attacks, which generally cause state estimation errors, have received widespread attention because these attacks can send inaccurate control signals to the executor [11–14]. Moreover, the critical detection technology for unknown attacks and unpredictable attack areas was proposed in the previous works [9, 15–20]. For instance, considering the characteristics of the network topology and transmission media, a risk prediction method based on a predictive model was proposed to accurately obtain the characteristics of the physical system, which can judge the fault area in the CPSs [16]. In [20], focusing on undetectable attacks, a dynamic attack detector was proposed.With the integration of multiple energy sources, the control platform and information transmission are extremely complicated [21]. Thus, other irresistible attacks, sensor, and actuator attacks are a topic of research. For example, in [22], a reliable control with the attack compensator was investigated which can withstand sensor and actuator attacks. In [1], the resilient control strategies were proposed to ensure that the variables converge to the equilibrium point in presence of sensor and actuator attacks.This paper concentrates on the study of an optimal robust control strategy, where the designed unified control method makes the power system immune to the actuator and sensor attacks. We use optimal control theory, reinforcement learning (RL), and neural networks (NNs) to design the controller under the assumed attacks of multiple characteristics. The main works and contributions can be summarized as follows:(1)
Optimal control theory, RL, and NNs are integrated to address the security issue of a benchmark power system.(2)
A unified way is proposed to deal with the sensor and actuator attacks via the optimal control design.(3)
The stability analysis with the NN approximation error, which is rarely discussed in the previous works, is studied in this paper.The rest of this paper is arranged as follows.First, the benchmark power system models with actuator and sensor attacks are formulated. Second, the optimal control issue for the nominal system is investigated, and the state-of-the-art RL methods along with the NN implementations are reviewed. Third, several robust control strategies are proposed for different types of cyberphysical attacks via the optimal control design, and stability proofs are derived through Lyapunov theory. Then, two different simulation examples demonstrate the effectiveness of our proposed methods. Finally, a brief conclusion is given.


## 2. Problem Statement for Power System

Let us consider the following benchmark power system:(1)Δψ˙f=−1TpΔψf+kpTpΔψt,Δψ˙t=−1TtΔψt+1TtΔψg,Δψ˙g=−1ksTgΔψf−1TgΔψg+1Tgu,where Δψf, Δψt, and Δψg represent the deviations of frequency, turbine power, and governor position value, respectively; Tt, Tg, and Tp denote the time constants of turbine, governor, and power system, respectively; kp and ks represent the gain of power system and the speed regulation coefficient, respectively; and u is the control input.Letx=Δψf,Δψt,ΔψgT. The nominal system (1) can be rewritten as(2)x˙=Ax+Bu,where A=−1/Tpkp/Tp00−1/Tt1/Tt−1/ksTg0−1/Tg and B=001/Tg.However, the attacks on the system are generally inevitable, which may affect the control performance. System dynamics (2) suffers from the actuator and sensor attacks, which can be, respectively, described by(3)x˙=Ax+Bu¯+Λ,(4)x˙=Ax+Bu¯+Λ,where u¯ is the robust control policy, which will be designed later. Λ denotes the system uncertainties. In this paper, we will consider different types of attacks.Due to the existence of unknown attacks, it is difficult or even impossible to investigate the systems (3) and (4) directly. Inspired by the idea of classical works [23–26], we convert this robust control issue of the systems (3) and (4) into the optimal control problem of the nominal system (2). The main idea is that, with the system data and models, we can first attain the optimal control policy through ADP algorithms. Subsequently, based on the optimal control form, we can develop different robust control strategies for the systems with various attacks.


## 3. Optimal Control for the Nominal System

Define the performance index function as(5)Jx0,u=∫0∞rxτ,uτdτ,where rx,u=xTQx+uTRu with positive definite symmetric matrices Q and R. Given the admissible control policy ux, the value function is expressed as(6)Vxt=∫t∞rxτ,uxτdτ.The optimal value function can be defined as(7)V∗xt=minu∫t∞rxτ,uxτdτ.According to the stationarity condition [27], the optimal control policy is derived by(8)u∗x=−12R−1BT∇V∗x,where ∇V∗x=∂V∗x/∂x and V∗x should satisfy the following Hamilton–Jacobi–Bellman (HJB) equation.(9)0=rx,u∗x+∇V∗TxAx+Bu∗x.Thus, the key point to obtain the optimal control policy is to solve the HJB equation.ADP is a powerful tool to solve the optimal control problems. Traditional ADP methods include two iterative algorithms: policy iteration (PI) and value iteration (VI). Afterwards, two noniterative RL methods are developed.


### 3.1. Online RL method

The aforementioned iterative ADP methods belong to the offline learning field because the value function and control policies are updated with the iteration index. Quite different from offline algorithms, online RL methods [27, 28] do not involve any iteration processes, and the value function and control policies are updated in real time.


### 3.2. Event Trigger-Based RL Method

In the online RL methods, the update and delivery of information must be continuous, which causes a waste of communication resources. For this phenomenon, the event trigger-based RL methods [29, 30] are developed. Here, the value function and control policies are updated only once when the system state error reaches the set point, which reduces the communication burden.By using the aforementioned ADP methods, we can obtain the optimal control form of the nominal system, which will be employed in the following sections.To implement the proposed algorithms, a critic NN and an actor NN are employed to approximate the iterative value function and control policy:(10)V^ix=ϕcTxWci,(11)u^ix=ϕaTxWai,where ϕcx and ϕax denote the NN activation functions and Wci and Wai represent the NN weights.Hence, the optimal value function and control policy have NN representation as(12)V∗x=ϕcTxWc,(13)u∗x=ϕaxWa,where Wc and Wa denote the ideal NN weights.In previous works, the NN approximation error was rarely discussed. In this paper, we attempt to consider its effect in the stability analysis.In Figure1, the sensor attacks, tampering the state values collected by sensors, occur in the sensor and communication network. Meanwhile, the actuator attacks, which generally modify the control instructions in actuator, occur between the decision and physical layer. The changed system state and control command can be eliminated by the robust control strategy which is calculated by RL based on the performance index function. Ultimately, the power system can work at the scheduled operating point under the sensor and actuator attacks.Figure 1
Block diagram of proposed robust control.


## 3.1. Online RL method

The aforementioned iterative ADP methods belong to the offline learning field because the value function and control policies are updated with the iteration index. Quite different from offline algorithms, online RL methods [27, 28] do not involve any iteration processes, and the value function and control policies are updated in real time.


## 3.2. Event Trigger-Based RL Method

In the online RL methods, the update and delivery of information must be continuous, which causes a waste of communication resources. For this phenomenon, the event trigger-based RL methods [29, 30] are developed. Here, the value function and control policies are updated only once when the system state error reaches the set point, which reduces the communication burden.By using the aforementioned ADP methods, we can obtain the optimal control form of the nominal system, which will be employed in the following sections.To implement the proposed algorithms, a critic NN and an actor NN are employed to approximate the iterative value function and control policy:(10)V^ix=ϕcTxWci,(11)u^ix=ϕaTxWai,where ϕcx and ϕax denote the NN activation functions and Wci and Wai represent the NN weights.Hence, the optimal value function and control policy have NN representation as(12)V∗x=ϕcTxWc,(13)u∗x=ϕaxWa,where Wc and Wa denote the ideal NN weights.In previous works, the NN approximation error was rarely discussed. In this paper, we attempt to consider its effect in the stability analysis.In Figure1, the sensor attacks, tampering the state values collected by sensors, occur in the sensor and communication network. Meanwhile, the actuator attacks, which generally modify the control instructions in actuator, occur between the decision and physical layer. The changed system state and control command can be eliminated by the robust control strategy which is calculated by RL based on the performance index function. Ultimately, the power system can work at the scheduled operating point under the sensor and actuator attacks.Figure 1
Block diagram of proposed robust control.


## 4. Robust Control Strategies for Actuator Attacks

First, let us consider the system (3) with Λ=Γx, where Γx≤kdx. The robust controller is designed by(14)u¯x=u∗x,where the parameters for generating u∗x will be determined later.Theorem 1.
If the positive definite matricesQ and R are selected appropriately, then system (3) is asymptotically stable under the robust controller (14).Proof.
Choose the Lyapunov function candidate as follows:(15)V=V∗x,which, according to (9), implies(16)V˙=V˙∗x=∇V∗TxAx+Bu∗x+Γx=∇V∗TxAx+Bu∗x+∇V∗TxBΓx=−xTQx−u∗TxRu∗x+∇V∗TxBΓx.
Substituting (8) into (16) yields(17)V˙=−xTQx−14∇V∗TxBR−1BT∇V∗x+∇V∗TxBΓx≤−xTQx+12ΓTxΓx−14∇V∗TxBR−1BT∇V∗x+12∇V∗TxBBT∇V∗x≤−λminQ−12kd2x2−14λminR−1−12BT∇V∗x2,where λmin⋅ denotes the minimum eigenvalue of a matrix.
To guaranteeV˙≤0, one should choose the parameters Q and R to satisfy the following inequalities:(18)λminQ≥12kd2,λminR−1≥2.
The proof is completed.Remark 1.
By using ADP methods, one can obtain the approximate optimal control policy. However, these ADP methods are finally implemented by NNs or other universal approximators, which will bring approximation errors. In the previous works, NN approximation errors were rarely discussed. In this paper, we attempt to present the corresponding error analysis.
When NNs finish learning, NN weights will achieve convergence. Based on (11), the NN-based approximate optimal control policy, which is actually applied to the system, is expressed as(19)u^x=ϕaxW^a,where W^a is the estimation of the ideal NN weight Wa. Let the approximation error be W˜a=W^a−Wa with W˜a≤W˜am.
By means of (13) and (19), one gets(20)u^x=ϕaxWa+ϕaxW^a−ϕaxWa=u∗x+ϕaxW˜a.Corollary 1.
If the positive definite matricesQ and R are selected appropriately, then the system (3) is asymptotically stable under the NN-based approximate optimal controller (19) as the NN weight approximation error W˜a goes to zero.Proof.
Utilizing the Lyapunov function candidate (15) yields(21)V˙=∇V∗TxAx+Bu^x+Γx=∇V∗TxAx+Bu∗x+Γx+∇V∗TxBϕaxW˜a.
Through the result of (17), equation (21) becomes(22)V˙≤−λminQ−12kd2x2−14λminR−1−12BT∇V∗x2+12BT∇V∗x2+12ϕaxW˜a2≤−λminQ−12kd2−12ka2W˜am2x2−14λminR−1−1BT∇V∗x2,where ϕax≤kax. If V˙≤0, the following condition should be satisfied:(23)λminQ≥12kd2+12ka2W˜am2,λminR−1≥4.
It can be observed that if the NN weight approximation errorW˜a goes to zero or is small enough, condition (23) can be easily realized with the chosen parameters. That is, the NN-based approximate optimal control policy can stabilize system (3).


## 5. Robust Control Strategies for Sensor Attacks

In this section, the proposed robust control schemes are modified and extended to deal with sensor attacks [31].


### 5.1. Extension to Nonlinear Sensor Attacks

Consider the system (4) with nonlinear sensor attacks:(24)x˙=Ax+Bu¯+Γx.The robust controller for (24) is designed the same as (14), i.e., u¯x=u∗x.Corollary 2.
If the matrixQ is selected appropriately, then the system (24) is asymptotically stable under the robust controller u¯x.Proof.
Choose the Lyapunov function candidate as (15). Then, one attains(25)V˙=∇V∗TxAx+Bu∗x+Γx=−xTQx−u∗TxRu∗x+∇V∗TxΓx≤−xTQx−u∗TxRu∗x+12ΓTxΓx+12∇V∗Tx∇V∗x≤−λminQ−12kd2−12kv2x2−u∗TxRu∗x,where ∇V∗x≤kvx.
To guaranteeV˙≤0, select the matrix Q to satisfy the following inequality:(26)λminQ≥12kd2+12kv2.
The proof is completed.
Note that the robust load frequency control problem is a special case of sensor attacks.
Let (24) be rewritten as(27)x˙=Ax+Bu¯+ΔPdt,where ΔPdt denotes the disturbance caused by the load demand change with ΔPdt≤ΔPdm.Corollary 3.
If the matrixQ is selected appropriately, then the system states of (27) are uniformly ultimately bounded under the robust controller u¯x.According to (15), one gets(28)V˙=∇V∗TxAx+Bu∗x+ΔPdt≤−λminQ−12kv2x2−u∗TxRu∗x+12ΔPdm2.Letz=xu∗x,Θ=λminQ−1/2kv2Ix00R with the identity matrix Ix, and cm=1/2ΔPdm2. If there exists a matrix Q which guarantees Θ to be positive definite, then (28) can be rewritten as(29)V˙≤−xu∗xTλminQ−12kv2Ix00Rxu∗x+12ΔPdm2≤−λminΘz2+cm.From (29), it can be observed that V˙≤0 if z≥cm/λminΘ. That is, the system states are uniformly ultimately bounded according to the Lyapunov extension theorem [27, 32, 33].


### 5.2. Extension to Constant Sensor Attacks

Consider the system (2) with constant sensor attacks:(30)x˙=Ax+Bu¯−Ξ,where Ξ˙=0.Letu¯=u∗ and add an attack compensator Ξ^ to (30). Then, (30) becomes(31)x˙=Ax+Bu∗+Ξ^−Ξ,where Ξ^˙=β−xT−∇V∗TxT.Theorem 2.
If the positive definite matricesQ and R are selected appropriately, then the system (31) is asymptotically stable under the optimal controller and the attack compensator.Proof.
Construct a Lyapunov function candidate as follows:(32)V=12xTx+V∗x+12βΞ˜TΞ˜,where Ξ˜=Ξ−Ξ^. Then, one has(33)V˙=xTAx+Bu∗−Ξ˜+∇V∗TxAx+Bu∗−Ξ˜−1βΞ^˙TΞ˜=xTAx+xTBu∗−xT+∇V∗TxΞ˜+∇V∗TxAx+Bu∗−1ββ−xT−∇V∗TxΞ˜.
After some mathematical derivation, equation (33) becomes(34)V˙=xTAx+xTBu∗−xTQx−u∗TRu∗≤Amx2+12x2+12Bm2u∗2−λminQx2−λminRu∗2=−λminQ−Am−12x2−λminR−12Bm2u∗2,where A≤Am and B≤Bm.
To ensureV˙≤0, one should set the parameters Q and R to satisfy the following inequalities:(35)λminQ≥Am+12,λminR≥12Bm2.
This completes the proof.
When NNs finish learning, the approximate optimal value function can be acquired:(36)V^x=ϕcTxW^c,where W^c is the estimation of the ideal NN weight Wc. Let the approximation error be W˜c=W^c−Wc with W˜c≤W˜cm.
Based on (31) and (36), the NN-based robust control scheme should be designed by(37)x˙=Ax+Bu^+Ξ^−Ξ=Ax+Bu∗x+BϕaxW˜a+Ξ^−Ξ,where(38)Ξ^˙=β−xT−∇V^TxT=β−xT−∇V∗Tx−W˜cT∇ϕcxT.Corollary 4.
If the positive definite matricesQ and R are selected appropriately, then the NN-based robust control scheme can stabilize the system (37) as the NN weight approximation errors W˜a and W˜c go to zero.Proof.
Employing the Lyapunov function candidate (32) yields(39)V˙=xTAx+Bu∗+ϕaxW˜a−Ξ˜+∇V∗TxAx+Bu∗+ϕaxW˜a−Ξ˜−1ββ−xT−∇V∗Tx−W˜cT∇ϕcxΞ˜=xTAx+xTBu∗+∇V∗TxAx+Bu∗+x+∇V∗xTBϕaxW˜a+W˜cT∇ϕcxΞ˜≤−λminQ−Am−12x2−λminR−12Bm2u∗2+x+∇V∗xTBϕaxW˜a+W˜cT∇ϕcxΞ˜.
From (39), in the limit as the NN weight approximation errors go to zero, i.e., W˜a⟶0 and W˜c⟶0, one can easily set the parameters Q and R to guarantee V˙≤0. If the NN approximation errors are not small enough or NNs fail to approximate optimal values, the result may not be asymptotically stable and the robust control scheme may be invalid. Therefore, the design of ADP learning algorithm is the key point.


## 5.1. Extension to Nonlinear Sensor Attacks

Consider the system (4) with nonlinear sensor attacks:(24)x˙=Ax+Bu¯+Γx.The robust controller for (24) is designed the same as (14), i.e., u¯x=u∗x.Corollary 2.
If the matrixQ is selected appropriately, then the system (24) is asymptotically stable under the robust controller u¯x.Proof.
Choose the Lyapunov function candidate as (15). Then, one attains(25)V˙=∇V∗TxAx+Bu∗x+Γx=−xTQx−u∗TxRu∗x+∇V∗TxΓx≤−xTQx−u∗TxRu∗x+12ΓTxΓx+12∇V∗Tx∇V∗x≤−λminQ−12kd2−12kv2x2−u∗TxRu∗x,where ∇V∗x≤kvx.
To guaranteeV˙≤0, select the matrix Q to satisfy the following inequality:(26)λminQ≥12kd2+12kv2.
The proof is completed.
Note that the robust load frequency control problem is a special case of sensor attacks.
Let (24) be rewritten as(27)x˙=Ax+Bu¯+ΔPdt,where ΔPdt denotes the disturbance caused by the load demand change with ΔPdt≤ΔPdm.Corollary 3.
If the matrixQ is selected appropriately, then the system states of (27) are uniformly ultimately bounded under the robust controller u¯x.According to (15), one gets(28)V˙=∇V∗TxAx+Bu∗x+ΔPdt≤−λminQ−12kv2x2−u∗TxRu∗x+12ΔPdm2.Letz=xu∗x,Θ=λminQ−1/2kv2Ix00R with the identity matrix Ix, and cm=1/2ΔPdm2. If there exists a matrix Q which guarantees Θ to be positive definite, then (28) can be rewritten as(29)V˙≤−xu∗xTλminQ−12kv2Ix00Rxu∗x+12ΔPdm2≤−λminΘz2+cm.From (29), it can be observed that V˙≤0 if z≥cm/λminΘ. That is, the system states are uniformly ultimately bounded according to the Lyapunov extension theorem [27, 32, 33].


## 5.2. Extension to Constant Sensor Attacks

Consider the system (2) with constant sensor attacks:(30)x˙=Ax+Bu¯−Ξ,where Ξ˙=0.Letu¯=u∗ and add an attack compensator Ξ^ to (30). Then, (30) becomes(31)x˙=Ax+Bu∗+Ξ^−Ξ,where Ξ^˙=β−xT−∇V∗TxT.Theorem 2.
If the positive definite matricesQ and R are selected appropriately, then the system (31) is asymptotically stable under the optimal controller and the attack compensator.Proof.
Construct a Lyapunov function candidate as follows:(32)V=12xTx+V∗x+12βΞ˜TΞ˜,where Ξ˜=Ξ−Ξ^. Then, one has(33)V˙=xTAx+Bu∗−Ξ˜+∇V∗TxAx+Bu∗−Ξ˜−1βΞ^˙TΞ˜=xTAx+xTBu∗−xT+∇V∗TxΞ˜+∇V∗TxAx+Bu∗−1ββ−xT−∇V∗TxΞ˜.
After some mathematical derivation, equation (33) becomes(34)V˙=xTAx+xTBu∗−xTQx−u∗TRu∗≤Amx2+12x2+12Bm2u∗2−λminQx2−λminRu∗2=−λminQ−Am−12x2−λminR−12Bm2u∗2,where A≤Am and B≤Bm.
To ensureV˙≤0, one should set the parameters Q and R to satisfy the following inequalities:(35)λminQ≥Am+12,λminR≥12Bm2.
This completes the proof.
When NNs finish learning, the approximate optimal value function can be acquired:(36)V^x=ϕcTxW^c,where W^c is the estimation of the ideal NN weight Wc. Let the approximation error be W˜c=W^c−Wc with W˜c≤W˜cm.
Based on (31) and (36), the NN-based robust control scheme should be designed by(37)x˙=Ax+Bu^+Ξ^−Ξ=Ax+Bu∗x+BϕaxW˜a+Ξ^−Ξ,where(38)Ξ^˙=β−xT−∇V^TxT=β−xT−∇V∗Tx−W˜cT∇ϕcxT.Corollary 4.
If the positive definite matricesQ and R are selected appropriately, then the NN-based robust control scheme can stabilize the system (37) as the NN weight approximation errors W˜a and W˜c go to zero.Proof.
Employing the Lyapunov function candidate (32) yields(39)V˙=xTAx+Bu∗+ϕaxW˜a−Ξ˜+∇V∗TxAx+Bu∗+ϕaxW˜a−Ξ˜−1ββ−xT−∇V∗Tx−W˜cT∇ϕcxΞ˜=xTAx+xTBu∗+∇V∗TxAx+Bu∗+x+∇V∗xTBϕaxW˜a+W˜cT∇ϕcxΞ˜≤−λminQ−Am−12x2−λminR−12Bm2u∗2+x+∇V∗xTBϕaxW˜a+W˜cT∇ϕcxΞ˜.
From (39), in the limit as the NN weight approximation errors go to zero, i.e., W˜a⟶0 and W˜c⟶0, one can easily set the parameters Q and R to guarantee V˙≤0. If the NN approximation errors are not small enough or NNs fail to approximate optimal values, the result may not be asymptotically stable and the robust control scheme may be invalid. Therefore, the design of ADP learning algorithm is the key point.


## 6. Simulation Example

In this section, to verify the proposed robust control strategy, two simulation examples of power systems are presented for two different types of attacks, respectively.


### 6.1. Design against Actuator Attacks

In this case, the actuator attack affecting the controller is considered in power system. The values of system parameters for this simulation are given in Table1. Then, we can obtain system matrices A=−1200−11−50−1 and B=001.Table 1
Values of system parameters.
ParametersTgTtTpkskpValues1110.22Let the initial system state values bex0=0.2;−0.3;0.4. When we insert the actuator attacks Λ=Γx into the system, the states of power systems become unstable, which is shown in Figure 2.Figure 2
System states under actuator attacks.Figure 3
2D plot ofΔψf and Δψt.In this case, the parameters are selected asQ=10I3 and R=0.2, respectively. One can attain the optimal control u∗x via Matlab command CARE or other RL methods. By using the robust controller (14), the system states under actuator attacks can be stabilized within 6 seconds in Figure 2. The 2D plot of state convergence trajectory is given in Figure 3, which indicates the nice performance of our control design.


### 6.2. Design against Constant Sensor Attacks

In this case, the designed controller is proved by numerical simulation results that it can effectively resist the sensor attacks. The values of system parameters for this simulation are given in Table2.Table 2
Values of system parameters.
ParametersTgTtTpkskpValues0.50.50.511Then, we can obtain system matricesA=−2200−22−20−2 and B=002. The controller parameters are selected as Q=8I3 and R=5. Let the initial system state values be x0=0.2;−0.2;0.4 and constant sensor attack be Ξ=0,0,0T0s≤t≤5s2,−2,1T5s<t≤15s−2,2,−1T15s<t≤30s.First, we present the simulation result without the attack compensator in Figure4, which implies the system states affected by constant attacks converge to unexpected values. Then, we employ the attack compensator-based robust control scheme, and simulation results are obtained in Figure 5, which shows the proposed scheme can quickly stabilize the system after the attacks occur. Figure 6 displays the dynamics of the attack compensator. Compared with the given attack, the compensator can estimate the attack value in a short time, which indicates the compensator can successfully get rid of the impact caused by the constant sensor attacks.Figure 4
System states without the attack compensator.Figure 5
System states with the attack compensator.Figure 6
Dynamics of the attack compensator.


## 6.1. Design against Actuator Attacks

In this case, the actuator attack affecting the controller is considered in power system. The values of system parameters for this simulation are given in Table1. Then, we can obtain system matrices A=−1200−11−50−1 and B=001.Table 1
Values of system parameters.
ParametersTgTtTpkskpValues1110.22Let the initial system state values bex0=0.2;−0.3;0.4. When we insert the actuator attacks Λ=Γx into the system, the states of power systems become unstable, which is shown in Figure 2.Figure 2
System states under actuator attacks.Figure 3
2D plot ofΔψf and Δψt.In this case, the parameters are selected asQ=10I3 and R=0.2, respectively. One can attain the optimal control u∗x via Matlab command CARE or other RL methods. By using the robust controller (14), the system states under actuator attacks can be stabilized within 6 seconds in Figure 2. The 2D plot of state convergence trajectory is given in Figure 3, which indicates the nice performance of our control design.


## 6.2. Design against Constant Sensor Attacks

In this case, the designed controller is proved by numerical simulation results that it can effectively resist the sensor attacks. The values of system parameters for this simulation are given in Table2.Table 2
Values of system parameters.
ParametersTgTtTpkskpValues0.50.50.511Then, we can obtain system matricesA=−2200−22−20−2 and B=002. The controller parameters are selected as Q=8I3 and R=5. Let the initial system state values be x0=0.2;−0.2;0.4 and constant sensor attack be Ξ=0,0,0T0s≤t≤5s2,−2,1T5s<t≤15s−2,2,−1T15s<t≤30s.First, we present the simulation result without the attack compensator in Figure4, which implies the system states affected by constant attacks converge to unexpected values. Then, we employ the attack compensator-based robust control scheme, and simulation results are obtained in Figure 5, which shows the proposed scheme can quickly stabilize the system after the attacks occur. Figure 6 displays the dynamics of the attack compensator. Compared with the given attack, the compensator can estimate the attack value in a short time, which indicates the compensator can successfully get rid of the impact caused by the constant sensor attacks.Figure 4
System states without the attack compensator.Figure 5
System states with the attack compensator.Figure 6
Dynamics of the attack compensator.


## 7. Conclusions

This paper has integrated optimal control theory, RL, and NNs to address the robust control issues of a benchmark power system. The optimal control theory for nominal systems and state-of-the-art RL methods along with the NN implementations have been reviewed. Multiple types of attacks in power systems, such as actuator attacks, nonlinear sensor attacks, and constant sensor attacks, are discussed. Then, several robust control schemes have been designed for different types of attacks, respectively. The control parameters have been derived through the Lyapunov stability theory. Furthermore, the stability analysis with the NN approximation error, which is rarely discussed in the previous works, has been presented in this paper. Simulation results have demonstrated the effectiveness of our proposed schemes.



---

*Source: 6646799-2021-03-23.xml*