## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning
### Analysis of Power Grid Stability and Convergence of Convex Neural Network Weights  
Equation (31) is used to learn the parameters of both actor networks and critic networks. Equation (22) satisfies the stability requirements of the aggregated distributed energy grids. When the parameters \(\beta\) and \(\gamma\) and the maximum absolute value of the eigenvalue \(\lambda_{max}(\mathbf{L}_{i})|\) of the matrix \(\mathbf{L}_{i}\) are selected appropriately, the proposed control strategy can achieve good control performance. In the following stability analysis, the overall system stability is indirectly proved by analyzing the subsystem stability. In the analysis of subsystem stability, the interconnection influence between subsystems is considered. If all the subsystems are stable, the overall system is stable. Thus, the stability of the subsystems is consistent with the stability of the overall system. The consideration of interconnection influence in the discussion of subsystem stability is reflected in the following four aspects. * The expected control output \(u_{d,i}\) designed in this paper and the gain term \(M_{i}\sum_{j\in D_{i}}T_{ji}\) of the physical interconnection disturbances included in the subsystem model (1) are involved. * The expected output of subsystem control \(u_{d,i}\) and the calculation of actual control output including the variable set \(\mathbf{x}_{\Lambda^{\gamma}(i)}\) of information adjacent subsystems are involved. * The modeling and estimation error term \(\chi_{i}\) in model (9) include the estimation error relationships of physical and information interconnection disturbances between subsystems. * When convex neural networks are used to approximate the critic networks and actor networks of reinforcement learning, the approximation errors \(\varepsilon_{a,i}\) and \(\varepsilon_{c,i}\) of convex neural networks are also considered. These two errors also include the estimation error relationships of physical and information interconnection disturbances between subsystems. In order to theoretically analyze the stability of the aggregated distributed energy grids and the convergence of the actor-critic network weights, the following assumptions are made. **Assumption 1**. : _The activation function \(\phi_{i}(. )\) is a non-decreasing and non-constant function that satisfies the global Lipschitz condition, and there is a positive number that satisfies the following inequality._  
\[0\leq\|\phi_{i}(\mathbf{W}_{i},\mathbf{x}_{i})-\phi_{i}(\mathbf{W}_{i}^{*},\mathbf{x}_ {i})\|\leq L_{i}^{\prime}\|\mathbf{W}_{i}-\mathbf{W}_{i}^{*}\|\]  
_where \(\mathbf{W}_{i}=\mathbf{W}_{a,i}\), \(\mathbf{W}_{i}^{*}=\mathbf{W}_{a,i}^{*}\), or \(\mathbf{W}_{i}=\mathbf{W}_{c,i}\), \(\mathbf{W}_{i}^{*}=\mathbf{W}_{c,i}^{*}\). The smallest possible \(L_{i}^{\prime}\) that satisfies the above inequality is called the Lipschitz constant of the function \(\phi_{i}(. )\)._  
**Note. ** For a non-decreasing and non-constant function, if there is a continuous derivative at any point in a fixed domain, the function must satisfy the Lipschitz condition in the domain. The activation function \(\phi_{i}(. )=\tanh x\) is a monotonically increasing function in a fixed continuous domain, and there is a continuous derivative at any point in the domain. Therefore, the activation function \(\phi_{i}(. )\) satisfies the Lipschitz condition. **Assumption 2**. : _Assume that the vectors \(\mathbf{w}_{a,i}\) and \(\mathbf{w}_{c,i}\) in the weight sets \(\mathbf{W}_{a,i}\) and \(\mathbf{W}_{c,i}\), and the control gain \(\mathbf{B}_{i}\) in the system model of aggregated distributed energy grids satisfy the following equation and inequality._  
\[\left\|\mathbf{w}_{a,i}{}^{\mathrm{T}}\mathbf{w}_{a,i}\right\|=\left\|\mathbf{w}_{c,i}{}^{ \mathrm{T}}\mathbf{w}_{c,i}\right\|=1,0\leq\|\mathbf{B}_{i,min}\|\leq\|\mathbf{B}_{i}\| \leq\|\mathbf{B}_{i,max}\|\]  
**Note. ** This assumption is reasonable.