## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning
### Critic Networks for Long Term Future Cost Approximation under DoS Attacks  
Critic networks approximate the long-term future cost \(Q_{i}(k)\) of the state trajectory under the current control strategy through the state information of power grids. The long-term future cost \(Q_{i}(k)\) is the weighted sum of the instantaneous cost at each future moment. The instantaneous cost is a function of the current system state and control output. This function not only considers the excessive frequency deviation but also measures the control cost. The long-term future cost output \(\hat{Q}_{i}(k)\) that evaluates the network approximation is the enhanced signal of reinforcement learning. Actor networks learn network parameters through the enhanced signal to reduce the control cost as much as possible and optimize the frequency control strategy by minimizing \(\hat{Q}_{i}\). The long-term future cost \(Q_{i}(k)\) of the state trajectory can be defined as follows. \[Q_{i}(k) =\alpha^{S}p_{i}(k+1)+\alpha^{S-1}p_{i}(k+2)+\ldots\alpha^{k+1}p_{ i}(S)+\ldots \tag{12}\]  
where \(\alpha\in(0,1)\) is the decay rate, and \(S\in Z+\) is a given positive integer for the long-term evaluation time window. \(p_{i}(k)\) is the instantaneous cost function, and its value is related to the number of violations of the given frequency deviation threshold and the size of the optimal control output [38], which is defined as follows. \[p_{i}(k)=\left\{ \begin{array}{ll}0,&a_{1}\|\mathbf{x}_{i}(k)\|+a_{2}\|u_{i}(k) \|\leq c\\ 1,&\textit{otherwise}\end{array} \right. \tag{13}\]  
where \(\|\bullet\|\) represents the 2-norm, \(a_{1}\) and \(a_{2}\) are the weights of the state and strategy utility functions, respectively, \(a_{1}+a_{2}\)=1, and \(c\) is the given cost threshold. When \(p_{i}=0\), the power grids is in a good performance state at the \(k\)-th moment; when \(p_{i}=1\), the power grids are in a bad performance state. Therefore, the control objective can be converted to the minimization of the long-term future cost \(Q_{i}(k)\). Under the optimal state trajectory, an iterative formula for \(Q_{i}\)[38] is shown as follows. \[Q_{i}(k)=\alpha Q_{i}(k-1)-\alpha^{S+1}p_{i}(k) \tag{14}\]  
In the control process, \(Q_{i}(k)\) can be used as an enhanced signal to optimize the control output of the ADES controller of the \(i\)-th subsystem. However, \(Q_{i}(k)\) cannot be directly analyzed and calculated without an analytical model. Therefore, this paper adopts approximation-learning based on convex neural networks. In order to improve the optimal search speed and efficiency of the learning algorithm, critic networks are constructed to approximate \(Q_{i}(k)\). The critic networks can be expressed as follows. \[\hat{Q}_{i}(k)=g_{i}(\mathbf{x}_{i}(k),u_{i}(k)) \tag{15}\]where \(\hat{Q}_{i}(k)\) is the approximate value of \(Q_{i}(k)\) and \(g_{i}(. )\) is the convex neural network function. However, \(\mathbf{x}_{i}\) may be disturbed by DoS attacks, which may not be normal state information. In order to solve this problem, this paper adopts an approximate approach [53] to solve the impact of DoS attacks on the system state. The critic network will use the non-attacked state information at the previous moment to replace the attacked state information at the current moment.This approach can reasonably eliminate the impact of DoS attacks. Suppose there is a set of weights \(\mathbf{W}_{c,i}\), \(g_{i}(. )\) that can be expressed as follows. \[g_{i}(.)=w_{c,i}h_{c,N}=Con(\mathbf{W}_{c,i}\mathbf{x}^{\prime}_{i}) \tag{16}\]  
where \(\mathbf{W}_{c,i}=\left\{\mathbf{W}_{c,0:N-1}^{(z)},\mathbf{W}_{c,0:N-1}^{(y)},\mathbf{w}_{c,i },\mathbf{b}_{c,0:N-1}\right\}\). The objective function of training \(g_{i}\) can be constructed based on the TD-error of \(\hat{Q}_{i}(k)\) derived from Equation (15) under the optimal trajectory. The TD-error can be expressed as follows.