## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning
### Critic-Actor Network Weight Learning  
In order to make convex neural networks be able to approximate \(u_{d,i}(k)\) and \(Q_{i}(k)\) to the greatest extent (minimize the objective functions \(E_{a,i}\) and \(E_{c,i}\)), the weight sets of both actor networks and critic networks need to be trained. \(\mathbf{W}_{c,i}\) and \(\mathbf{W}_{a,i}\) need to be updated to the optimal weight sets \(\mathbf{W}_{c,i}^{*}\) and \(\mathbf{W}_{a,i}^{*}\). According to \(u_{i}(k)\), \(\hat{Q}_{i}(k)\) and the objective functions (18) and (25), the online learning algorithm of the weight set of convex neural networks can be obtained. \(\sqrt{\mathbf{B}_{i}}^{\mathrm{T}}\sqrt{\mathbf{B}_{i}}=\mathbf{1}^{\mathrm{T}}\mathbf{B }_{i}\) is defined, where \(\mathbf{1}=[1,1,1,1,1,1]^{\mathrm{T}}\). Thus, Equation (25) can be expanded as follows. \[E_{a,i}(k)=\frac{1}{2}\mathbf{e}_{a,i}(k)^{\mathrm{T}}\mathbf{e}_{a,i}(k)=\frac{1}{2} \mathbf{1}^{\mathrm{T}}\mathbf{B}_{i}\Delta u_{i}^{2}(k)+\frac{1}{2}\hat{Q}^{2}( k-1) \tag{27}\]  
where \(\Delta u_{i}(k)=u_{i}(k)-u_{d,i}(k)\). However, some parameters in the expected control output \(u_{d,i}(k)\) are unknown; therefore, \(\Delta u_{i}(k)\) cannot be directly expressed by this formula. Thus, \(\mathbf{B}_{i}\Delta u_{i}(k)\) is indirectly expressed by Equation (1). \(\mathbf{B}_{i}\Delta u_{i}(k)=\mathbf{x}_{i}(k+1)-\mathbf{L}_{i}\mathbf{x}_{i}(k)-\mathbf{\chi}_{i}\). For the objective function of critic network (18), it is expanded as follows. \[E_{c,i}(k) =\frac{1}{2}e_{c,i}(k)^{\mathrm{T}}e_{c,i}(k)=\frac{1}{2}\Big{[} \hat{Q}_{i}(k)-\alpha(\hat{Q}_{i}(k-1)-\alpha^{S}p_{i}(k))\Big{]}^{2}=\frac{1 }{2}\hat{Q}_{i}^{2}(k)+\frac{1}{2}\alpha^{2}\hat{Q}_{i}^{2}(k-1)\] \[\quad+\frac{1}{2}\alpha^{2(S+1)}p_{i}^{2}(k)-\alpha\hat{Q}_{i}(k) \hat{Q}_{i}(k-1)-\alpha^{S}p_{i}(k)\hat{Q}_{i}(k)+\alpha^{S+1}p_{i}(k)\hat{Q} _{i}(k-1) \tag{28}\]  
The update process of the weight sets \(\mathbf{W}_{a,i}(k)\) and \(\mathbf{W}_{c,i}(k)\) depends on the gradient descent algorithm. However, only using the gradient descent method to update the weights of actor-critic networks cause some problems. Since \(\mathbf{W}_{i}(k)\) needs to meet certain constraints, if only the gradient descent method is used in the learning process to update the weights of actor-critic networks, the updated network weights may not be able to maintain within the constraint set. Therefore, it is necessary to ensure that the weight set after the gradient descent update always falls in the feasible region (non-negative weight). The projected gradient algorithm is used to ensure that the weight constraints are maintained. The projection of matrix \(\mathbf{W}^{\prime}\) on the constraint set \(\Omega\) is defined as follows. \[\Pi_{\Omega}(\mathbf{W}^{\prime})=\operatorname*{arg\,min}_{\mathbf{W}\in\Omega }\frac{1}{2}\
orm{\mathbf{W}-\mathbf{W}^{\prime}}_{F}^{2} \tag{29}\]  
where \(\Omega\) is the constraint set of the weights of convex neural networks, \(\Omega=\left\{\mathbf{W}_{i}|\mathbf{W}_{1:N-1}^{(z)}\geq\mathbf{0}\right\}\). Given the initial weight state \(\mathbf{W}_{1:N-1}^{(z)}(0)\in\Omega\), the learning rates \(0<\beta<1\) and \(0<\gamma<1\) of actor-critic network weights, the projected gradient descent method extends the standard gradient descent to a feasible set \(\Omega\). Therefore, based on the projected gradient descent method, the update rules of \(\mathbf{W}_{a,i}(k)\) and \(\mathbf{W}_{c,i}(k)\) at time \(k\) can be expressed as follows. \[\mathbf{W}_{a,i}(k+1)=\mathbf{W}_{a,i}(k)-\beta\Pi_{\Omega}\bigg{(}\frac{ \partial E_{a,i}(k)}{\partial\mathbf{W}_{a,i}(k)}\bigg{)},\mathbf{W}_{c,i}(k+ 1)=\mathbf{W}_{c,i}(k)-\gamma\Pi_{\Omega}\bigg{(}\frac{\partial E_{c,i}(k)}{ \partial\mathbf{W}_{c,i}(k)}\bigg{)} \tag{30}\]  
According to Equations (27) and (28), \(\partial E_{a,i}(k)/\partial\mathbf{W}_{a,i}(k)\) and \(\partial E_{c,i}(k)/\partial\mathbf{W}_{c,i}(k)\) can be calculated. Therefore, the update rules of the actor-critic network weight set can be re-expressed as follows.