# Deep-Reinforcement-Learning-Based Joint Energy Replenishment and Data Collection Scheme for WRSN
## 4 A Joint Energy Replenishment and Data Collection Algorithm Based on Deep Reinforcement Learning
### UAV Control Algorithm Based on MODDPG  
In contrast with reinforcement learning algorithms based on an action value, our scenario involves a continuous action space for controlling the instantaneous speed and heading angle of the UAV. This necessitates a distinct modeling and solution approach. To tackle this continuous action space, we propose utilizing the MODDPG model for policy learning. The algorithmic specifics of MODDPG are elaborated in Algorithm 2. ```
0: The coordinates of CH nodes \(l_{i}\)
0: The sequence of CH nodes \(l_{new}\)
1: Initialize maximum iteration temperature \(K_{0}=50,000\), the sequence of CH nodes \(l_{i}\), and annealing coefficient \(q=0.98\). 2:\(K\gets K_{0}\). 3:while\(K>K_{0}\)do
4: Swap the order of any two nodes to generate a new visiting sequence \(l_{new}\). 5: Calculating \(l_{new}\), denoted as \(df\). 6:if\(df>0\)then
7:if\(exp(\frac{df}{K_{0}})>rand(0,1)\)then
8:\(l_{f}\gets l_{new}\)
9:endif
10:else
11:\(K=K\times q\)
12:endif
13:endwhile
```  
**Algorithm 1** Solve TSP by using simulated annealing algorithm  
Figure 3: Illustration of the algorithm process. ```
0: State space \(s_{rl}\)
0: The action of UAV \(a_{rl}\)
1: Initialize Actor network parameters \(\theta^{\mu}\), and Target Actor network parameters \(\theta^{\mu^{\prime}}\), \(\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}\). 2: Initialize Critic network parameters \(\theta^{Q}\), and Target Critic network parameters \(\theta^{Q^{\prime}}\), \(\theta^{Q^{\prime}}\leftarrow\theta^{Q}\). 3: Initialize experience Replay Buffer. 4:for episode \(=0\) to \(M\)do
5: Initialize \(s_{rl}\). 6:for time step=\(t_{1},t_{2},\dots,t_{n}\)do
7:repeat
8: With probability of \(\epsilon\) choose an action \(a_{rl}=clip(\mu(s_{rl}|Q_{\mu})+\epsilon,a_{low},a_{high})\). 9: Perform action \(a_{rl}\) and observe reward \(r_{rl}\) and next state \(s^{\prime}_{rl}\). 10: Store the transition \((s_{rl},a_{rl},r_{rl},s^{\prime}_{rl})\) from \(P\). 11:if\(P\geq\) Batch size then
12: Randomly sample Mini batch transitions \((s_{rl},a_{rl},r_{rl},s^{\prime}_{rl})\) from \(P\). 13: Compute \(y_{rl}\) (19). 14: Update Critic network by minimizing the critic loss (20). 15: Update Actor network by maximizing the actor loss (21). 16: Soft Update Target Network Parameters. 17:\(\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\) \(\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}\)
18:endif
19:until\(\sum_{n=0}^{N}t_{n}\geq T\)
20:endfor
21:endfor
```  
**Algorithm 2** MODDPG  
* State Space: The state space set is defined as follows: \[s_{rl}(t)=\left\{d^{x}_{u,ch_{rl}^{latr}}(t),d^{y}_{u,ch_{rl}^{latr}}(t),x_{u}(t ),y_{u}(t),N_{f}(t),N_{d}(t),\forall i\in[1,m]\right\}\] (28) Specifically, with a state space size of 6, \(d^{x}_{u,ch_{rl}^{latr}}\) and \(d^{y}_{u,ch_{rl}^{latr}}\) represent the relative distances in the horizontal and vertical coordinates between the UAV and the target CH node \(i\), and \(N_{f}(t)\) records the cumulative number of times that the UAV has continuously exceeded the restricted area by time \(t\). * Action Space: At each \(t\), the action of the UAV is defined as follows: \(\theta_{u}(t)\in(0,2\pi]\): Instantaneous heading angle of the UAV along the horizontal direction at \(t\). \(v_{u}(t)\in[0,v_{max}]\): Instantaneous speed of the UAV at \(t\). The action of the UAV is defined as \(a_{rl}=[v_{u}(t),\theta_{u}(t)]\)
* Reward Function: At any time step \(t_{n}\), the UAV receives a reward \(r_{rl}\). To ensure the throughput of the UAV, the UAV receives an immediate reward \(r_{0}\) whenever it successfully establishes communication with the target node \(ch_{l}^{latr}\). The reward function is defined as follows: \[r_{rl}= \begin{cases}r_{0}(t)+r_{1}(t),\text{ if }\Delta_{u,ch_{l}^{latr}}(t)\leq R _{d}\\ r_{1}(t),\text{otherwise}\end{cases}