# Deep-Reinforcement-Learning-Based Joint Energy Replenishment and Data Collection Scheme for WRSN
## 4 A Joint Energy Replenishment and Data Collection Algorithm Based on Deep Reinforcement Learning
### UAV Control Algorithm Based on MODDPG
\] (29) \[r_{0}(t)=r_{serve}+\eta_{1}\lambda_{\text{cover}}(t)+\eta_{2}R_{ch_{l}^{latr} }^{train}(t)\] (30) \[r_{1}(t)=-\eta_{2}\Delta_{u,ch_{l}^{latr}}(t)-\eta_{3}(E_{u}^{pro}(t)+N_{d}(t))\] (31)Specifically, the reward function is divided into two parts, \(r_{0}\) and \(r_{1}\), where \(r_{serve}\) represents the reward for the UAV establishing a connection with CH nodes; \(\lambda_{cover}\) indicates the number of nodes covered by the UAV; and \(\eta_{1}\), \(\eta_{2}\), and \(\eta_{3}\) are reward weight factors for the optimization objectives. \(r_{1}\) is based on the UAV actions at each time step and includes factors such as the distance between the UAV and CH nodes, the energy consumption of the UAV flight, and the number of dead nodes \(N_{d}(t)\). The MODDPG network framework is shown in Figure 4; it consists of four networks, with the main network and target network each comprising two networks: the actor network, critic network, target actor network, and target critic network. The main network and target network share the same network structure, where the actor network outputs UAV actions, and the critic network evaluates these actions to update the actor network. The MODDPG algorithm optimizes the UAV instantaneous speed and heading angles, ultimately learning the optimal policy. The calculation of the target value \(y_{rl}\) is as follows:  
\[y_{rl}=r_{rl}+\gamma Q^{\prime}({s_{rl}}^{\prime},\mu^{\prime}({s_{rl}}^{ \prime};\theta^{\mu^{\prime}});\theta^{Q^{\prime}}) \tag{32}\]  
The critic network is trained using gradient descent to minimize the loss between the predicted value \(Q(s_{rl},a_{rl};\theta^{Q})\) and the target value \(y_{rl}\). \[Loss=\frac{1}{M}{\sum{(y_{rl}-Q(s_{rl},a_{rl};\theta^{Q}))}^{2}} \tag{33}\]  
Update the actor network using a gradient-based policy algorithm:  
\[\
abla_{\theta^{\mu}}J=\frac{1}{M}\sum{\
abla_{a_{rl}}Q(s_{rl},a_{rl};\theta^{ Q})|_{s=s_{rl},a=\mu(s_{rl})}\
abla_{\theta^{\mu}}\mu(s_{rl};\theta^{\mu})|_{s_{rl}}} \tag{34}\]  
where \(J\) represents the total discounted cumulative reward. The specific algorithm flow is described as follows: Initialize the replay buffer, critic network, and actor network parameters \(\theta^{Q}\) and \(\theta^{\mu}\), as well as the target critic network and target actor network parameters \(\theta^{Q^{\prime}}\) and \(\theta^{\mu^{\prime}}\) (lines 1-3). Perform the exploration phase for the UAV in lines 4-10, and generate an action \(a_{rl}=clip(\mu(s_{rl}|Q_{\mu})+\epsilon)\) from the actor network. This process is repeated until the maximum task time is reached, and the experiences are stored in the replay buffer. Updating network parameters, when the experience replay buffer \(P\) is full, start training the networks to update actor network and critic network parameters (lines 11-15). Update the critic network parameters by minimizing the loss function (line 14), and update the actor network parameters using the deterministic policy gradient (line 15). Finally, use the  
Figure 4: MODDPG network framework. soft update technique to update the parameters of the target actor network, controlling the update frequency to converge faster toward the optimal policy (lines 17-21).