## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning
### Actor Networks for Control Strategy under DoS Attacks  
Actor networks are used to calculate the control output \(u_{i}(k)\) of the ADES controller. The input of the actor networks includes the state information of the local subsystem \(i\) and the remote network connection subsystem \(j(j\in\mathcal{N}^{\prime}(i))\). This information is used for the learning of the actor networks and the calculation of the control output. The control output acts on the power electronics interface of power grids to eliminate sudden frequency fluctuations in power grids. The evaluation signal output by the critic networks (also called the reinforced signal of reinforcement learning) can be used to select the optimal control output, but it may be difficult to stabilize the dynamic online learning process of power grids when the selection of the optimal control output only relies on the reinforced signal. Therefore, an expected control output is needed to measure the closeness between the output of the actor networks and the optimal control strategy of power grids. According to the dynamic characteristics of power grids, the expected control output \(u_{d,i}(k)\) is assumed to be used to improve the exponential stability and short-term performance of the system. The expected control output makes the power grids present certain stable dynamic characteristics and ensures that the state \(\mathbf{x}_{i}(k)\) of power grids approaches 0, and thus the dynamic characteristics of the power grids have the following form. \[\mathbf{x}_{i}(k+1)=\mathbf{L}_{i}\mathbf{x}_{i}(k)+\mathbf{B}_{i}[u_{i}(k)-u_{d,i}(k)]+\mathbf{ \chi}_{i} \tag{19}\]  
The expected control output \(u_{d,i}(k)\) is the optimal controllable active power injected by the subsystem \(i\). The frequency stability of the power grids is maintained through the balance of active power and reactive power. \(u_{d,i}\) is expressed as follows. \[u_{d,i}(k) =-d_{f_{i}}\Delta f_{i}\Big{[}1+M_{i}\sum_{j\in D_{i}}T_{ji}\Big{]} -((M_{i}-D_{i}\Delta t)/\Delta t)\Delta P_{mi}\] \[-((M_{i}+\Delta t)/\Delta t)P_{vi}-\sum_{j\in\mathcal{N}(i)\cap \mathcal{N}^{\prime}(i)}\Delta P_{ji} \tag{20}\]  
where \(d_{f_{i}}\) is a hypothetical positive real number. \(u_{d,i}\) can satisfy the eigenvalue of \(\mathbf{L}_{i}\) existing in the unit circle so that the state equation of the system is changed to Equation (1). If the error between \(u_{i}\) and \(u_{d,i}\) is bounded, and the error \(\chi_{i}\) is bounded, the power grids are bounded and stable. \[\chi_{i}=\sum_{j\in\mathcal{N}(i)\setminus\mathcal{N}^{\prime\prime}(i)}\delta_{ ji}\mathbf{B}_{ji}\chi_{j}(k)+\mathbf{E}_{i}\varpi_{i}(k) \tag{21}\]  
According to the parameter \(d_{f_{i}}\), the specific form of \(\mathbf{L}_{i}\) can be expressed as Equation (22). \[\mathbf{L}_{i}=\left[ \begin{array}{cccccc}0&1-\Delta t\sum_{j\in D_{i}}T_{ji} &0&0&0&0\\ 1+\frac{\Delta t}{M_{i}}-\frac{d_{f_{i}}\Delta t}{M_{i}}&1&0&0&0&0\\ -d_{f_{i}}\sum_{j\in D_{i}}T_{ji}\Delta t&1&0&0&0\\ 0&0&1-\frac{\Delta t}{T_{ji}}\frac{\Delta t}{T_{ji}}&0&1-\frac{\Delta t}{T_{ ji}}&1-\frac{K_{i}\Delta t}{T_{ji}}&0\\ 1&1-b_{i}\Delta t&0&0&0&0\\ 0&0&0&0&0&1-\frac{\Delta t}{T_{ji}}\end{array} \right] \tag{22}\]  
The control system focuses on reducing the frequency deviation \(\Delta f_{i}\) to zero or constrains the change of \(\Delta f_{i}\) to a fixed small interval as much as possible. However, due to the difficulty of analyzing and modeling a large system, such as power grids, traditional methods cannot directly calculate the expected control output \(u_{d,i}(k)\). The approximation of the approximated \(u_{d,i}(k)\) is denoted as \(u_{i}(k)\). In order to further optimize the long-term performance of the control strategy, this paper adds a reinforced signal \(\hat{Q}_{d,i}(k)\) as one of the evaluation signals to measure the control strategy. An additional evaluation error signal \(\hat{Q}_{i}(k)-\hat{Q}_{d,i}(k)\) of the strategy utility function can be regarded as the reinforced signal of the critic networks to the actor networks.