## 4 Deep Deterministic Policy Gradient  
Frequency limit controller (FLC) is a common DC modulation method in practical engineering [22; 23], and its control block diagram is shown in Figure 1. When the system fails and the system frequency oscillation deviation exceeds the upper or lower limit of the FLC dead zone, the FLC is automatically activated and the DC power is automatically adjusted by a closed-loop PI controller. \begin{table}
\begin{tabular}{c c c c c c} \hline DC Modulation Sequence & 1 & \(\dots\) & \(m\) & \(\dots\) & \(l\) \\ \hline name & 1 & \(\dots\) & \(m\) & \(\dots\) & \(l\) \\ DC modulation sensitivity & \(\varphi_{\mathrm{max}}\) & \(\dots\) & \(\varphi_{m}\) & \(\dots\) & \(\varphi_{\mathrm{min}}\) \\ \hline \end{tabular}
\end{table} 
Table 1: DC modulation sequence table. A deep deterministic policy gradient algorithm (DDPG, Figure 2) is proposed to rectify the PI parameters of FLC. The Deep Deterministic Policy Gradient method uses the Actor-Critic algorithm as its basic framework, approximates the policy and action-value functions using a deep learning network, and uses the stochastic gradient method to train the parameters in both the policy network and value network models. A dual neural network model of real-time network and target network is used for both policy and value functions. At the same time, the algorithm borrows the experience playback mechanism from the DQN algorithm, in which the experience data generated by the interaction between the Actor and the environment will be stored in the experience pool. Then, a batch of data samples will be extracted for training so that the algorithm can be converged more easily. At each time period \(t\), the agent observes the current state \(s_{t}\), performs an action \(a_{t}\), receives a reward value \(r_{t}\) from the environment \(E\), and then transitions to the next state \(s_{t+1}\) according to the state transition probability function. For FLC, which adjusts the system frequency, the integral of the absolute value of the system frequency deviation and the product of time is selected as the reward function of DDPG optimization algorithm--that is, the optimization objective function is as follows:  
\[\min J=\int_{0}^{T}t|\Delta f|dt. \tag{25}\]  
The state space \(s_{t}\) is:  
\[s_{t}=(t,K_{P},K_{I},\Delta f,\Delta P), \tag{26}\]  
the state space includes the current period \(t\), PI parameters \(K_{P}\), and \(K_{I}\) of DC FLC. \(\Delta f\) is the frequency deviation, and \(\Delta P\) is the active power deviation. Action space \(a_{t}\) is:  
\[a_{t}=(\Delta K_{P},\Delta K_{I}). \tag{27}\]  
Figure 1: Block diagram of FLC. Figure 2: DDPG algorithm framework. The reward function \(r_{t}\) is:  
\[r_{t}=-\int_{0}^{T}t|\Delta f|dt. \tag{28}\]  
The first part of the reward function is chosen as the inverse of the integral of the absolute value of the system frequency deviation and the product of time. The greater the reward the agent receives, the smaller the integral of the absolute value of the system frequency deviation and the time product. The loss function is calculated according to the following formula, where \(y_{t}\) is the target \(Q\) value. \[\left\{ \begin{array}{l}y_{t}=r_{t}+\gamma Q^{\prime}(s_{t+1},\pi^{\prime}(s_ {t+1}\Big{|}\theta^{\pi^{\prime}})\Big{|}\theta^{Q^{\prime}})\\ L(\theta^{Q})=E\left(\gamma_{t}-Q(s_{t},a_{t}\big{|}\theta^{Q})\right)^{2} \end{array} \right.. \tag{29}\]  
According to the gradient update rule, the current value network is updated by calculating the gradient of the loss function [24]:  
\[\left\{ \begin{array}{l}\theta_{k}^{Q}=\theta_{k-1}^{Q}-\mu_{Q}\overline{ \boldsymbol{V}}_{\theta^{Q}}L(\theta_{k-1}^{Q})\\ \overline{\boldsymbol{V}}_{\theta^{Q}}L(\theta_{k-1}^{Q})=E(2(y_{t}-Q(s,a \Big{|}\theta_{k-1}^{Q})\Big{|}_{s=s_{t},a=a_{t}})\\ \overline{\boldsymbol{V}}_{Q\theta^{Q}}(s,a\Big{|}\theta_{k-1}^{Q})\Big{|}_{ s=s_{t},a=a_{t}})\end{array} \right. \tag{30}\]  
The policy network uses a function of the value network output as a loss function.