# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 4 PDDQN-HHVBF Protocol
### Selection of the Optimal Relay Node  
This part will specifically introduce how to select a relay node for forwarding based on the DDQN algorithm based on empirical priority after the candidate relay node set is determined above. The PDDQN-HHVBF routing protocol provides a complete execution process for the data packets from the AUV to the Sink node. In this paper, it is assumed that nodes can obtain their own residual energy and three-dimensional coordinates in M-UWSNs. In addtion, all nodes in M-UWSNs will drift due to the movement of ocean currents, but the moving distance is limited in a short time, that is, the network topology is stable relatively in a short time. Compared with the long routing propagation time in the underwater acoustic channel and the high energy consumption, the delay and energy consumption of calculating the \(Q\) value are very low, so it is not calculated separately in this paper. The process of requesting the \(Q\) value is shown in Figure 9. \begin{table}
\begin{tabular}{c c} \hline \hline
**Experimental Parameters** & **Values** \\ \hline Transmission range & 100 m \\ Initial energy & 10,000 J \\ Transmission power & 10 W \\ Receiving power & 1 W \\ Control packet power & 0.2 W \\ Number of nodes & 500–3000 \\ AUV speed & 10 m/s \\ Node random movement speed & 0–3 m /s \\ Packet size & 20 Bytes \\ Transmission rate & 1 plt/s \\ Batch sample size & 256 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Experimental parameters. Figure 8: The schematic diagram of SumTree. As shown in Figure 9, the pre-forwarding node requests the \(Q\) value of the candidate relay nodes before sending data, and the request packet carries the location information of the pre-forwarding node so as to calculate the location parameter of the \(Q\) value. After receiving the request, candidate relay nodes calculates the \(Q\) value and returns them to the pre-forwarding node. After receiving the \(Q\) value of the candidate relay nodes, the Source node selects the node with the largest \(Q\) value as the relay node to transmit data, that is, node 2 in Figure 9. Each node in M-UWSNs can be regarded as an agent. The position information, energy and number of candidate relay nodes of each sensor node can be regarded as the current state. The forwarding of the relay node selected by the sending node can be regarded as the action taken by the agent according to the state, and a \(Q\) value will be obtained each time the data packet is forwarded to the relay node. In this protocol, a data packet needs to obtain the \(Q\) value of the candidate relay node, so it first sends a request \(Q\) value packet to all candidate relay nodes, and the candidate relay node calculates the \(Q\) value and returns the value to the sending node. The sending node selects the candidate relay node with the largest \(Q\) value to forward the data, and then the chosen node forwards the data to the Sink node hop by hop according to this method. When the node calculates the \(Q\) value, it needs to obtain the residual energy, location information, and the number of candidate relay nodes. The formula of residual energy is as Formula (15). The location information is \(\alpha\), in reference [13], as shown in Formula (16). The number of candidate relay nodes is \(Num_{can}\), which is the number of candidate relay nodes. \[r_{\text{sen}}=\frac{e_{\text{own}}}{e_{\text{max}}} \tag{15}\]  
\[\alpha=\frac{\left(R-d*\sin\theta_{j}\right)}{R} \tag{16}\]  
in Formula (15), \(e_{\text{own}}\) is the existing energy of the node, \(e_{\text{max}}\) is the initial energy of the node. In Formula (16), as Figure 3, \(R\) refers the radius of the pipeline, \(d\) is the distance that the pre-forwarding node deliveries data to the relay node, and \(\theta_{j}\) represents the angle between \(\left|\overrightarrow{n_{i}n_{j}}\right|\) and \(\left|\overrightarrow{n_{i}Sink}\right|\). Figure 9: The schematic diagram of the Q value of pre-forwarding node request.