## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning  
This paper proposes a power grid frequency control strategy of convex neural network-based reinforcement learning to solve the following three issues. (1) Communication networks of ADES controller among subareas suffer from DoS attacks. (2) Analytical modeling issues occur in the frequency control system design of large-scale and complex power grids. (3) The issues involve the fast optimization of a power grid frequency control strategy.  
The actor-critic online reinforcement-learning structure is used to achieve both ADES-based power grid frequency control and disturbance suppression of DoS attacks. Convex neural networks are used to construct both actor and critic networks in the actor-critic structure algorithm. Convex neural networks can convert the nonlinear optimization problems in the critic networks into approximated convex relaxed optimization problems to avoid local optimization in the optimization process, and ensure that the actor networks can output suboptimal control output.  
In addition, the convexity of the convex neural networks enables the training process of the actor networks to reach the convex point quickly as the gradient drops. In this case, the convergence speed of network weights is higher than that of general neural networks. The overall workflow of the power grid frequency control strategy of convex neural network-based reinforcement learning is shown in Figure 2.  
In Figure 2, the frequency controller of each subsystem consists of both actor networks and evaluation networks. The critic networks are used to approximate the reinforced signal \(Q_{i}(k)\), and the actor networks are used to approximate the optimal control strategy. The reinforced signal \(\hat{Q}_{i}(k)\) is used to optimize the actor networks, thereby, improving the control output \(u_{i}(k)\).  
Considering the impact of DoS attacks on communications, the proposed control strategy focuses on achieving the following two goals. (1) This control strategy can quickly dampen frequency fluctuations caused by sudden disturbances so that the power grid frequency deviation can be eliminated in a short time and so that the stability of power grids can be maintained. (2) This frequency control strategy can effectively resist the interference caused by DoS attacks.  
Therefore, this paper adopts a load-frequency control strategy of convex neural network-based online reinforcement learning and uses BESS to eliminate the frequency deviation of power grids under DoS attacks. Additionally, online reinforcement learning can monitor the changes in the state of distributed power grid in real time, eliminate useless  
Figure 2: The overall work-flow of the power grid control strategy.  
data blocked by DoS attacks, adjust the control output in real time, and provide the optimal control output for the state control of distributed energy grids, thereby, realizing real-time defense against DoS attacks.  
The actor-critic structure is applied to the implementation of reinforcement learning. This paper introduces a convex neural network to approximate actor networks and critic networks. The introduction of convex neural network is shown below.