## 4 Materials and Methods  
### Deep Deterministic Policy Gradient (DDPG) Approach  
As explained in the Introduction Section, the DDPG is an optimal candidate for wind turbine generator control. The DDPG algorithm is a model-free, online, off-policy reinforcement learning method that utilizes an actor-critic reinforcement learning agent that can search for an optimal policy that can maximize the expected cumulative long-term reward [20]. The DDPG algorithm utilizes two neural networks known as actor-critic networks, where the actor, denoted as \(\pi(S;\theta)\) with parameters \(\theta\), takes an observation, S, and returns the corresponding action that maximizes the long-term reward [20]. The critic, denoted as \(Q(S,A;\phi)\) with parameters \(\phi\), takes an observation, S, and an action, A, and returns the long-term expected value [20]. The DDPG also leverages a target actor and a target critic. The target actor, denoted as \(\pi_{t}(S;\theta_{t})\), is used to improve the stability of the optimization by allowing the agent to periodically update the target actor parameters, \(\theta_{t}\), using the latest actor parameter values [20]. The target critic, denoted as \(Q_{t}(S,A;\phi_{t})\), is also used to stabilize the optimization, as the agent periodically updates the target critic parameters, \(\phi_{t}\), using the latest critic parameter values [20].  
The training algorithm, depicted on MathWorks [20], begins by initializing the critic with random parameter values and then initializes the target critic with the same parameter values. This is also done with the actor and target actor parameters, respectively. For each training time step, there are 8 steps that will be completed, which are listed as follows [20]:  
1. For the current observation S, select an action A = \(\pi(S;\theta)+N\), where N is the stochastic noise.
2. Execute action A. Observe the reward and next observation \(S^{\prime}\).
3. Store the experience (\(S,A,R,S^{\prime}\)) in the experience buffer.
4. Sample a random mini batch of M experiences (\(S_{i}\), \(A_{i}\), \(R_{i}\), \(S^{\prime}_{i}\)) from the experience buffer.
5. If \(S^{\prime}_{i}\) is a terminal state, set the value function target \(y_{i}\) to \(R_{i}\). Otherwise, set it to Equation (15). \[y_{i}=R_{i}+\gamma Q_{t}\big{(}S^{\prime}_{i},\pi_{t}\big{(}S^{\prime}_{i}; \theta_{t}\big{)};\phi_{t}\big{)}\] (15)The value function target is the sum of the experience reward, \(R_{i}\), and the discount future reward. To compute the cumulative reward, the agent first computes the next action by passing the next observation, \(S^{i}_{i}\), from the sampled experience to the target actor. The agent then finds the cumulative reward by passing the next action to the target critic [20].  
1. Update the critic parameters by minimizing the loss, L, across all sampled experiences, as depicted in the following Equation (16). \[L=\frac{1}{2M}\sum_{i=1}^{M}(y_{i}-Q(S_{i},\ A_{i};\phi))^{2}\] (16)
2. Update the actor parameters using the following sampled policy gradient to maximize the expected discounted reward, as depicted in Equations (17)-(19). \[\
abla_{\theta}J\ \approx\ \frac{1}{M}\sum_{i=1}^{M}G_{ai}G_{\pi i}\] (17) \[G_{ai}=\Delta_{A}Q(S_{i},\ A_{i};\phi)\ \ \ \ \ \text{where}\ A=\pi(S_{i};\theta)\] (18) \[G_{\pi i}=\
abla_{\theta}\pi(S_{i};\theta)\] (19)
3. Update the target actor and critic parameters depending on the target method.  
The target update method has 3 options: smoothing, periodic, and periodic smoothing [20].  
Smoothing--Update the target parameters at every time step using the smoothing factor, \(\tau\). The equations for updating the target parameters utilizing the smoothing factor are depicted in Equations (20) and (21).  
\[\phi_{t} =\tau\phi+(1-\tau)\phi_{t}\ (\text{for critic parameters}) \tag{20}\] \[\theta_{t} =\tau\theta+(1-\tau)\theta_{t}\ (\text{for actor parameters}) \tag{21}\]  
Periodic--Update the target parameters periodically without smoothing.  
Periodic Smoothing--Update the target parameters periodically with smoothing.