## 3 Optimized Load Frequency Control with Convex Neural Network-Based Reinforcement Learning
### Convex Neural Network Structure Design  
Finding the optimal control problem is converted to getting the control output or strategy that minimizes the long-term future cost \(Q_{i}(k)\). However, the general direct search process for the long-term future cost \(Q_{i}(k)\) is a non-convex problem. The existence of multiple local sub-optimal control strategies may not only lead to the inability to find the global optimal control strategy but also reduce the speed of optimization and convergence.  
Therefore, this paper uses convex neural networks to build an approximated model of \(Q_{i}(k)\). The optimization of reinforcement learning is carried out in the approximated model established by convex neural networks so that the optimization process of reinforcement learning is approximated as the convex optimization process.  
In ordinary machine learning, convex generally means that the parameters are convex, and thus the weights are convex [40; 52]. The convexity of convex neural networks means that the output is convex with respect to the input, while the network parameters are not fully convex. To make the output of convex neural networks be a convex function about the input, the convex neural networks need to meet the following constraints. (1) The weight matrix \(\mathbf{W}_{1:N-1}^{(z)}\) of the middle layer of convex neural networks must be positive semi-definite. (2) All activation functions \(\phi_{m-1}(.)\) of convex neural networks are convex non-decreasing functions.  
This paper defines a \(Q_{N}\)-layer neural network structure based on multiple inputs/outputs through recursive methods. The neural networks are called convex neural networks. The structure of the convex neural networks is shown in Figure 3. The convex neural networks consist of one input layer, \(N\) intermediate layers, and one output layer. The input layer is the state vector of each subsystem, which is denoted as \(\boldsymbol{x}_{i}^{\prime}=[\boldsymbol{x}_{1},\boldsymbol{x}_{2},\ldots, \boldsymbol{x}_{N}]^{\mathsf{T}}\).  
Suppose the dimension of \(\boldsymbol{x}_{i}(k)\) is \(l=6\). \(\hat{\mathbf{W}}_{m-1}^{(z)}\in\mathbf{R}^{Nl\times Nl}\) and \(\hat{\mathbf{W}}_{m-1}^{(y)}\in\mathbf{R}^{Nl\times Nl}\) are the weights of the middle layer, \(\boldsymbol{b}_{m-1}\in\mathbf{R}^{Nl\times 1}\) is the bias of the middle layer, \(\hat{h}_{m}\in\mathbf{R}^{Nl\times 1}\) is the output vector of the \(m\)-th middle layer, \(\delta\) is the output of the output layer, and \(\boldsymbol{w}_{i}\in\mathbf{R}^{1\times Nl}\) is the connection weight between the output layer and the last middle layer, \(m=1,2\ldots N\).  
Figure 3: The structure of the convex neural networks.  
The output \(\mathbf{h}_{m}\in\mathbf{R}^{N\times 1}\) of the \(m\)-th middle layer can be expressed as follows.  
\[\mathbf{h}_{m}=\phi_{m-1}(\mathbf{W}_{m-1}^{(z)}\mathbf{h}_{m-1}+\mathbf{W} _{m-1}^{(y)}\mathbf{x}_{i}^{\prime}+\mathbf{b}_{m-1})\] \[s.t.\mathbf{W}_{1:N-1}^{(z)}\geq\mathbf{0},\mathbf{W}_{0}^{(z)}\equiv \mathbf{0},\mathbf{h}_{0}\equiv\mathbf{0} \tag{10}\]  
where \(\phi_{m-1}(.)\) is the activation function \(\tanh x\) and \(\hat{\mathbf{W}}_{1:N-1}^{(z)}\geq 0\), \(\mathbf{W}_{0}^{(z)}\equiv\mathbf{0}\) and \(\mathbf{h}_{0}\equiv\mathbf{0}\) are the constraints. If and only if convex neural networks satisfy the above constraints, the output is the convex function of the input.  
Suppose there is a set of weights \(\mathbf{W}_{i}=\left\{\mathbf{W}_{0:N-1}^{(z)},\mathbf{W}_{0:N-1}^{(y)},\mathbf{w} _{i},\mathbf{b}_{0:N-1}\right\}\). Then, the output layer of convex neural networks with respect to the output of middle layers can be expressed as follows.  
\[o=\mathbf{w}_{i}\mathbf{h}_{N}=Con(\mathbf{W}_{i},\mathbf{x}_{i}^{\prime}) \tag{11}\]  
where \(Con(.)\) represents the mapping function between input and output.