## 3 Frequency Coordination Control Strategy at LCC Terminal
### AGC Controller Parameter Optimization Using DQN Algorithm  
The fundamental idea of deep learning (DL) is to transform high-dimensional data into a low-dimensional feature representation through multi-layer neural networks and nonlinear transformations. The fundamental idea of reinforcement learning (RL) is to interact with the environment in real time, learning from experiences in the process of continuous failures and successes, maximizing the cumulative reward obtained by the agent from the environment, and ultimately allowing the agent to learn the optimal strategy. The DQN algorithm effectively integrates deep learning and reinforcement learning, addressing the instability issues associated with using neural networks and other nonlinear function approximators to represent action-value functions. The DQN algorithm is based on Q-learning and uses a neural network model instead of a Q-value table to directly predict Q-function values. The current target Q-value is calculated, and then the neural network parameters are updated using the mean square error between the current target Q-value and the predicted Q-value of the neural network. Since AGC controllers are crucial for maintaining power system stability and operation, the optimization of AGC controller parameters involves a high-dimensional parameter space, which needs to handle the complex dynamic environment. Compared with other optimization algorithms, DQN optimization can handle the high-dimensional state space in many practical problems, and it can adapt to the changes in system dynamics and learn the optimal control strategy under different operating conditions. Therefore, we chose to use the DQN algorithm to optimize the PI parameters of the AGC controller. The core ideas of the DQN algorithm can be summarized in the following three points:  
(1) Loss Function. Training a neural network essentially involves optimizing a loss function. The lower the loss function, the higher the robustness of the model. The DQN algorithm constructs a learnable loss function based on the Q-learning algorithm. The update formula for the algorithm is shown in Equation (12):  
\[Q(s,a)\gets Q(s,a)+\text{lr}\bigg{(}r+\underset{a\in A}{\text{max}}Q \big{(}s^{\prime},a\big{)}-Q(s,a)\bigg{)}. \tag{12}\]  
We use the mean squared error between the current Q value and the target Q value to define the loss function of the DQN algorithm, as shown in Equation (13):  
\[L(\theta)=E\Bigg{[}\bigg{(}r+\gamma\underset{a\in A}{\text{max}}Q \big{(}s^{\prime},a^{\prime};\theta\big{)}-Q(s,a;\theta)\bigg{)}^{2}\Bigg{]}, \tag{13}\]  
where \(\theta\) represents the weight parameters of the convolutional neural network model, and \(\gamma\) represents the decay factor, \(r+\gamma\text{max}_{a\in A}Q(s^{\prime},a^{\prime};\theta)\), which is the target Q value. After obtaining the loss function, the weight parameters \(\theta\) of the convolutional neural network can be directly solved using the gradient descent method. (2) Experience Replay Mechanism. The DQN algorithm adopts an experience replay mechanism, storing the experiential data generated by the agent's interaction with the surrounding environment in an experience pool. During network iteration training, several data points are randomly extracted from the experience pool for model training. Each experiential data are stored in the form of a five-tuple \((s,a,r,s^{\prime},T)\), where \(T\) is a Boolean value used to indicate whether the next state \(s^{\prime}\) is a target or terminal state. This can effectively eliminate the correlation between data samples, reduce the error of approximating the value function, and avoid mutual influence between data and non-stationary distribution problems, thereby improving the convergence speed of the network model. Dual Network Structure. DQN adopts two neural networks with identical structures but different parameters, which are called action value estimation network and target value network.