## 3 Control and Structure Design  
### Control Design by Reinforcement Learning  
Before introducing control design, a cost function is defined to measure the control cost \(Q\). Control performance usually involves the quadratic of state and control output. Thus, we define the control cost \(Q\) by a control strategy utility function used in our previous work [24]:  
\[Q_{i}(k)=\min_{u_{i}(k+j),j\in[0,\infty]}\sum_{t=0}^{\infty}\alpha^{N-t}p_{i} (k+t), \tag{9}\]  
where \(0<\alpha<1\) is the discount rate, and \(N\) is a positive integer. \(p_{i}\) as a binary performance index is defined as:  
\[p_{i}(k)=\left\{ \begin{array}{ll}1,&a_{1}\left\|x_{i}(k)\right\|+a_{2}\left\| u_{i}(k)\right\|\leq c,\\ 0,&\text{otherwise},\end{array} \right. \tag{10}\]  
where \(\left\|\cdot\right\|\) denotes 2-norm, \(a_{1}\) and \(a_{2}\) are the weights of state and control cost respectively, and \(c\) is the threshold of performance indication. If the current cost is in an allowed range (less than \(c\)), the binary performance index shown in Equation (10) is 1; otherwise, it is 0. Binary performance index makes the strategy utility function limited. When time horizon is limited, even the system diverges, and it can avoid a numerical crash in the learning process. To prevent the system diverge, we construct a desired control output \(u_{di}\) to apply a damping rate \(L_{i}\in R^{5\times 5}\), \(\|L_{i}\|<1\) to the system, so that the power grid frequency dynamic becomes  
\[x_{i}(k+1)=L_{i}x_{i}(k)+\sum_{j\in N_{p}(i)\backslash N_{c}(i)}B_{ji}x_{j}(k)+ E_{i}w_{i}(k). \tag{11}\]  
If we approximate \(u_{di}\) by a neural network output \(\hat{u}_{di}\), the neural network is defined as  
\[\hat{u}_{di}(k)=M_{a}W_{i}\phi(k), \tag{12}\]  
where \(\phi\in R^{nl\times 1}\) is the radial basis of neural network calculated from state \(x_{1},x_{2},...,x_{n}\), \(W_{i}\in R^{nl\times nl}\) is the trainable weight, and \(M_{a}\in R^{1\times nl}\) is a given constant matrix. As shown in Figure 2, the proposed neural network structure is based on a radial basis function (RBF). An RBFb-based neural network can identify nonlinear dynamical systems [26]. \(l\) is the dimension of radius basis for one area. Under the control of \(\hat{u}_{di}=u_{i}\), the system of area \(i\) becomes  
\[x_{i}(k+1)=L_{i}x_{i}(k)+B_{i}[\hat{u}_{di}(k)-u_{di}(k)]+\sum_{j\in N_{p}(i) \backslash N_{c}(i)}B_{ji}x_{j}(k)+E_{i}w_{i}(k). \tag{13}\]  
According to the control theory of stability, if \(B_{i}[\hat{u}_{di}(k)-u_{di}(k)]+E_{i}w_{i}(k)\) and \(\sum_{j\in N_{p}(i)\backslash N_{c}(i)}B_{ji}x_{j}(k)\) in the right part of Equation (13) are bounded, the system is ultimately uniformly bound (UUB) [27]. We assume the disturbance \(w_{i}(k)\) is bounded as \(\|w_{i}(k)\|\leq\varepsilon\), where \(\varepsilon\) is a small real number, so learning \(W_{i}\) should involve the approximation of \(u_{di}\). The control cost involving system performance is also approximated by a neural network as shown in Equation (14):  
\[\hat{Q}_{i}(k)=M_{c}W_{i}\phi(k), \tag{14}\]  
where \(M_{c}\in R^{1\times nl}\) is a given constant matrix. We train the neural network weight \(W_{i}\) to approximate the desired control output \(u_{di}\) as well as the performance measurement for the improvement of system performance. Therefore, we define a loss function for training as  
Figure 2: Neural network structure. \[V_{i}(k)= \frac{1}{2}\left\|B_{i}\hat{a}_{di}(k)-B_{i}u_{di}(k)\right\|^{2}+ \tag{15}\] \[\frac{1}{2}\left\|\alpha^{N+1}p_{i}(k)+\alpha^{-1}\hat{Q}_{i}(k+1 )-\hat{Q}_{i}(k)\right\|^{2}+\frac{1}{2}\left\|\hat{Q}_{i}(k)\right\|^{2}.\]  
The first term denotes the approximation error of the desired control output, the second term denotes the approximation error of system performance, and the third term is used to improve the long-term system performance. As shown in Equation (16), the desired control output \(u_{di}\) and system performance \(Q_{i}\) can be expressed by the neural network:  
\[\left\{