## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning
### Actor Networks for Control Strategy under DoS Attacks
When the approximated control output approximates the expected control output, the reinforced signal gradually approaches zero, and thus \(\hat{Q}_{d,i}\) is set to 0. Therefore, the objective function of training \(\varphi(. )\) can be represented by the error composed of the control error \(\Delta u_{i}(k)\) and the reinforced signal \(Q_{i}(k)\). Suppose there is a set of weights \(\mathbf{W}_{a,i}\), \(\varphi_{i}(. )\) that can be expressed as follows. \[\varphi_{i}(.)=u_{lim}\tanh(w_{a,i}h_{a,N})=u_{lim}\tanh(Con(\mathbf{W}_{a,i}, \boldsymbol{x^{\prime}}_{i})) \tag{23}\]  
where \(\mathbf{W}_{a,i}=\left\{\mathbf{W}_{a,0;N-1}^{(z)},\mathbf{W}_{a,0;N-1}^{(y) },\mathbf{w}_{a,i},\mathbf{b}_{a,0;N-1}\right\}\), and \(u_{lim}\) is the maximum output constraint of \(u_{i}\). According to Equation (23), \(\varphi_{i}(. )\) is calculated by \(u_{lim}\tanh(. )\), and thus \(u_{i}\) is constrained to \([-u_{lim},u_{lim}]\). Therefore, after passing through the inertia link of the battery energy storage system, \(\Delta P_{B_{i},lim}\) can also satisfy the Equation (6). The following two subsections introduce the basic structure and implementation of both critic networks and actor networks. The error composed of the control error \(\Delta u_{i}(k)\) and the reinforced signal \(\hat{Q}_{i}(k-1)\) output by the critic networks is expressed as follows. \[\mathbf{e}_{a,i}(k)=\left[\sqrt{\mathbf{B}_{i}}\Delta u_{i}(k),\hat{Q}_{i}(k-1) \right]^{\mathrm{T}} \tag{24}\]  
where \(\Delta u_{i}(k)=u_{i}(k)-u_{d,i}(k)\) is the error value between the approximated control output and the expected control output. Therefore, according to the error \(\mathbf{e}_{a,i}(k)\), the objective function for policy network learning can be defined as follows. \[E_{a,i}(k)=\frac{1}{2}\mathbf{e}_{a,i}^{\mathrm{T}}(k)\mathbf{e}_{a,i}(k) \tag{25}\]  
The constant convexity of convex neural networks can ensure the existence of the global optimal solution in the reinforcement-learning optimization process [40], and prevent the optimization process from falling into the local optimum. Additionally, the robustness of convex neural networks is good when they are applied to solve optimization problems [41]. When power grids suffers from DoS attacks, the transmission of the state information \(\mathbf{x}_{j}\) of some remotely connected subsystems may be blocked. In the control process, both actor networks and critic networks deal with the blocked state information. The blocked state information is replaced by the state information of the previous moment rather than 0. Therefore, the estimation of the state information \(\mathbf{x}_{j}\) can be expressed as follows. \[\mathbf{\hat{x}}_{j}(k)=\left\{ \begin{array}{l}\mathbf{x}_{j}(k-1),\delta_{ji}=0,k>1 \\ \mathbf{x}_{j}(k),\delta_{ji}=1\end{array} \right. \tag{26}\]  
As a main reason, the state information of each subsystem cannot quickly become 0 during normal operations. When the state information of a subsystem changes too much which induced noneligible estimation errors, the actor-critic networks can predict the reinforced signal and control output with the estimation errors since the robustness of the neural networks [53]. The optimization of state information is completely completed by the actor-critic networks. The impact of DoS attacks on state information is eliminated by the actor-critic networks. The actor network predicts the current state quantity based on the past state value. The predicted value is corrected by actor network parameters and system measurement values. Therefore, the issues of DoS attacks can be solved by reinforcement learning.