# Deep-Reinforcement-Learning-Based Joint Energy Replenishment and Data Collection Scheme for WRSN
## 4 A Joint Energy Replenishment and Data Collection Algorithm Based on Deep Reinforcement Learning
### Cluster Head Selection Algorithm  
Referring to [39], this network studied in this paper is a UAV-WRSN that uses a dynamic routing protocol. Figure 2 illustrates the generated clusters and data flow. Compared with static routing protocols, dynamic routing protocols can evenly consume the energy of nodes. However, in a traditional LEACH routing protocol [45], the CM nodes determine whether to become CH nodes based on randomly generated thresholds. For each CM node \(cm_{j}\), \(w_{j}^{th}\) is given as follows:  
\[w_{j}^{th}=\left\{\begin{array}{ll}\frac{p}{1-p(r\bmod\frac{1}{p})},&\text{ if }j\in cm_{j}\\ 0,&\text{ otherwise}\end{array}\right. \tag{26}\]  
where \(p\) is the expected percentage of CH nodes, and \(r\) is the current number of rounds. This method lacks consideration for the remaining energy and location of the nodes. We have redesigned the CH node selection mechanism based on these two factors. Due to insufficient energy information before the network operation, we utilize the K-means algorithm to partition all the data into \(m\) clusters and identify \(m\) centroids within each cluster. The overall optimization objective of the K-means algorithm is to minimize the sum of distances between each point and its respective centroid within the clusters. Based on the distances between nodes and centroids, we have determined \(m\) CH nodes. As a result, when the UAV reaches a CH node, it can cover more nodes, thereby improving the charging efficiency.  
At each round, the cluster member nodes at each cluster elect CH nodes based on their own energy status and location information to balance the energy consumption of the nodes. Dynamic routing protocols are used within each cluster. We stipulate no communication link between the sensor nodes and BS. CM nodes use CH nodes as the next hop for communication, while CH nodes collect data sent by member nodes and transmit them to the UAV. (If a CH node has nonzero data storage after the UAV completes a flight cycle, it will continue to transmit data as a CM node in the next mission.) The weight of CM nodes can be obtained using the following formula:  
\[w_{cm_{j}}=\kappa\frac{E_{\text{max}}-RE_{j}^{cm}(t)}{RE_{\text{are}}(t)}+(1- \kappa)\frac{d_{i,j}}{d_{\text{are}}} \tag{27}\]  
Figure 2: Illustration of clusters and data flow.  
where \(RE_{ave}(t)\) represents the average remaining energy of CM nodes, \(d_{ave}\) represents the average distance between CM nodes and CH nodes, \(d_{ij}\) represents the distance between the CH node \(ch_{i}\) and the CM node \(cm_{j}\), and \(\kappa\) is a weighting coefficient. The weight value \(w_{j}\) represents the priority or importance of CM; among CM nodes, the one with the highest \(w_{j}\) will be selected as the CH node in the next mission round. The CH node will then join the cluster as a CM node. Based on the obtained coordinates of the CH nodes, this paper utilizes the SA algorithm to solve the TSP. The pseudocode for the SA algorithm is presented in Algorithm 1. The specific algorithmic process is shown in Figure 3.