# Energy-Efficient Power Scheduling Policy for Cloud-Assisted Distributed PV System: A TD3 Approach
## 4 Problem Solution
### The Solution Based on TD3
#### 4.2.1 DDPG Algorithm  
DDPG algorithm is a kind of reinforcement learning algorithm that can handle continuous action spaces, and it combines the advantages of Deep Q-Network (DQN) algorithm and Deterministic Policy Gradient (DPG) algorithm, and can effectively learn the optimal policy in high-dimensional and complex environments. DDPG algorithm is based on the Actor-Critic framework, which uses two deep neural networks to approximate the value function and the policy function, respectively. One is the Critic network, which evaluates the action value of the current state, and the other is the Actor network, which generates the optimal action of the current state. The objective of the Critic network is to minimize the mean squared error between the current action value and the target action value, while the objective of the Actor network is to maximize the action value given by the Critic network. To improve the stability and convergence of learning, DDPG algorithm also uses the techniques of Experience Replay (ER) and Target Network (TN), that is, in each iteration, a mini-batch of state-action-reward-next state tuples is randomly drawn from the replay buffer as training data, and a fixed target network is used to calculate the target action value, instead of using the real-time updated network. The procedure of DDPG algorithm is shown in Algorithm 1 and the architecture is shown in Figure 2.  
```
1: Initialize the Actor network \(\pi_{\phi}\) of and the Critic network \(Q_{\theta}\) with random weights \(\phi\) and \(\theta\), as well as the target Actor network \(\pi_{\phi^{{}^{\prime}}}\) and the target Critic network \(Q_{\phi^{{}^{\prime}}}\) with the same weights of the original network parameters, i.e., \(\phi^{{}^{\prime}}\leftarrow\phi,\theta^{{}^{\prime}}\leftarrow\theta\). 2: Initialize a replay buffer \(D\), which is used to store the sampled tuples \((s_{t},a_{t},r_{t},s^{\prime}_{t})\), where \(s_{t}\) is the state, \(a_{t}\) is the action, \(r_{t}\) is the reward, \(s^{\prime}_{t}\) is the next state in the MDP. 3:for each episode \(t=1\) to \(T\)do
4: Initialize a random process \(N\), which is used to add exploration noise to the Actor network. 5: Observe the initial state \(s_{1}\). 6:for each time step \(t\)do
7: Select an action \(a_{t}=\pi_{\phi}(s)+N_{t}\) according to the current policy and exploration noise. 8: Execute the action and observe the reward \(r_{t}\) and the next state \(s^{{}^{\prime}}\). 9: Store the transition tuple \((s_{t},a_{t},r_{t},s^{{}^{\prime}})\) in the replay buffer \(D\). 10: Randomly sample a mini-batch of tuples from the replay buffer \(D\), denoted as \((s_{i},a_{i},r_{i},s^{\prime}_{t})\), where \(i=1,2,...,N\) and \(N\) is the batch size. 11: For each sampled transition tuple, calculate the target action value \(y_{i}=r_{i}+\gamma Q_{\phi^{{}^{\prime}}}\left(s^{\prime}_{t},\pi_{\phi^{{}^{ \prime}}}\left(s^{\prime}_{t}\right)\right)\), where \(\gamma\) is the discount factor. 12: Update the critic network by minimizing the mean squared error loss, i.e., \(\theta\leftarrow\theta-\alpha\bigtriangledown\sigma_{n}\frac{1}{N}\sum_{i=1}^{N }\left(Q_{\theta}\left(s_{i},\tau_{\phi}(s_{i})\right)-y_{i}\right)^{2}\), where \(\alpha\) is the learning rate. 13: Update the actor network by using the sampled policy gradient, i.e., \(\phi\leftarrow\phi-\alpha\bigtriangledown_{\phi}\frac{1}{N}\sum_{i=1}^{N}Q_{ \theta}\left(s_{i},\tau_{\phi}(s_{i})\right)\). 14: Update the target networks by using soft updates, i.e., \(\theta^{{}^{\prime}}\leftarrow\tau\theta+(1-\tau)\theta^{{}^{\prime}}\), \(\phi^{{}^{\prime}}\leftarrow\tau\phi+(1-\tau)\phi^{{}^{\prime}}\), where \(\tau\) is the update rate. 15: If \(s^{\prime}_{t}\) is a terminal state, end the episode, otherwise, let \(t\gets t+1\), and continue the next time step. 16:endfor
```  
**Algorithm 1** DDPG  
However, DDPG algorithm also has some drawbacks and problems, mainly in the following two aspects:  
Figure 2: The structure of DDPG algorithm.