# Energy-Efficient Power Scheduling Policy for Cloud-Assisted Distributed PV System: A TD3 Approach
## 4 Problem Solution
### The Solution Based on TD3
#### 4.2.2 TD3 Algorithm  
Twin Delayed Deep Deterministic Policy Gradient (TD3) is an improved version of the DDPG algorithm, which addresses the issues of overestimation bias and policy instability in DDPG algorithm, and proposes three key techniques:  
* Twin critics: TD3 algorithm uses two Critic networks to evaluate the action value of the current state, denoted as \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\). When calculating the target action value, TD3 algorithm uses the minimum value of the two Critic networks, that is, \[y_{i}=r_{i}+\gamma\min_{i=1,2}Q_{\theta_{i}^{\prime}}\left(s_{i}^{\prime},\pi _{\phi^{\prime}}\left(s_{i}^{\prime}\right)\right)\] (14) where \(r_{i}\) is the reward, \(\gamma\) is the discount factor, \(Q_{\theta_{i}^{\prime}}\) and \(\pi_{\phi^{\prime}}\) are the target Critic network and the target Actor network, respectively. The purpose of this is to avoid overestimation bias, and improve the quality of the policy. * Delayed policy update: TD3 algorithm uses a delayed policy update mechanism, that is, only after the Critic network is updated d times, the Actor network and the target network are updated once. The purpose of this is to reduce the frequency of policy update, and reduce the variance of the policy, and improve the stability of the policy. * Target policy smoothing: TD3 algorithm uses a target policy smoothing technique, that is, when calculating the target action value, a random noise is added to the action generated by the target Actor network, that is, \[y_{i}=r_{i}+\gamma\min_{i=1,2}Q_{\theta_{i}^{\prime}}\left(s_{i}^{\prime},\pi _{\phi^{\prime}}\left(s_{i}^{\prime}\right)+\epsilon\right)\] (15) where \(\epsilon\sim N(0,\sigma)\) is a random noise that follows a normal distribution, \(\sigma\) is the standard deviation of the noise, \(\gamma\) is the discount factor, \(Q_{\theta_{i}^{\prime}}\) and \(\pi_{\phi^{\prime}}\) are the target Critic network and the target Actor network, respectively. The purpose of this is to prevent the policy from being too greedy, and increase the exploration of the policy, and improve the robustness of the policy. The process of TD3 algorithm is basically consistent with that of DDPG algorithm, except that the three techniques mentioned above are used when calculating the target action value, updating the Critic network, and updating the Actor network. The pseudocode of TD3 algorithm is shown in Algorithm 2, and the architecture diagram of TD3 algorithm is shown in Figure 3.  
```
1: Initialize the Actor network \(\pi_{\phi}\) of and the Critic network \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\), as well as the target Actor network \(\pi_{\phi^{\prime}}\) and the target Critic network \(Q_{\theta^{\prime}_{1}}\) and \(Q_{\theta^{\prime}_{2}}\), where the target network parameters are the same as the network parameters, i.e., \(\phi^{{}^{\prime}}\leftarrow\phi,\theta^{\prime}_{i}\leftarrow\theta_{i}\), \(i=1,2\). 2: Initialize a replay buffer \(D\), which is used to store the sampled tuples \((s_{t},a_{t},r_{t},s^{\prime}_{t})\), where \(s_{t}\) is the state, \(a_{t}\) is the action, \(r_{t}\) is the reward, \(s^{\prime}_{t}\) is the next state in the MDP..
3:for each episode \(t=1\) to \(T\)do
4: Initialize a random process \(N\), which is used to add exploration noise to the Actor network. 5: Observe the initial state \(s_{1}\) from the environment. 6:for each time step \(t\)do
7: According to the current Actor network, select an action \(a_{t}=\pi_{\phi}(s)+N(0,\sigma)\), where \(\sigma\) is the standard deviation of the noise and then execute the action. 8: Observe the reward \(r_{t}\) and the next state \(s^{{}^{\prime}}\). 9: Store the transition tuple \((s_{t},a_{t},r_{t},s^{{}^{\prime}})\) in the replay buffer \(D\). 10: Randomly sample a mini-batch of tuples from the replay buffer \(D\), denoted as \((s_{i},a_{i},r_{i},s^{\prime}_{i})\), where \(i=1,2,...,N\) and \(N\) is the batch size.