# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
### Motivation and Contributions  
Mobile source node Underwater Wireless Sensor Networks (M-UWSNs) is a network structure in which Autonomous Underwater Vehicle (AUV) is used as source node to collect data and transmit data packets hop-by-hop to Sink node through underwater nodes. Due to long propagation in underwater, there are high packet collision rate and unreliable transmission, it will lead to low packet delivery ratio (PDR), high energy consumption,  
Figure 1: The schematic diagram of M-UWSNs.  
short network lifetime in M-UWSNs. In addition, due to the small number of nodes and the distribution is scattered in the sparse network, M-UWSNs faces the problem of routing holes, which leads to low PDR and extra energy wasting. Specially for AUV with routing holes, energy limited AUV will waste a lot of energy in the data forwarding process, resulting in not enough energy for AUV to complete tasks.  
In view of the above problems, this paper proposes an improved HHVBF protocol based on the empirical priority DDQN algorithm, namely, the PDDQN-HHVBF (Empirical Priority DDQN to Improve Hop-by-Hop Vector-Based Forwarding) protocol. The protocol use Empirical Priority DDQN to find the optimal relay node in pipeline referred HHVBF (Hop-by-Hop Vector-Based Forwarding) protocol by requesting the Q value considering the number of candidate relay nodes, the residual energy of nodes, and the geographical location information of nodes from candidate relay nodes in time. The proposed protocol aims to the high PDR and the long network lifetime in the large-scale network, and effectively avoids the occurrence of routing holes caused by ocean current movement in sparse M-UWSNs. Finally, the \"Store-Carry-Forward\" mechanism proposed for AUV, this mechanism store and carry packets when facing routing holes, which will improve PDR and save energy of AUV significantly. The main contributions of this paper are as follows:  
1. We design the PDDQN-HHVBF routing protocol to find the optimal relay node to forward in M-UWSNs. The DDQN algorithm based on empirical priority has the advantages of fast convergence speed, fast learning speed and stable learning process. It can quickly converge in M-UWSNs and obtain the formula for finding the optimal relay node function.
2. PDDQN-HHVBF enables the pre-forwarding nodes find the optimal relay nodes with the max Q value to propagate packets considering the the geographical location information of nodes in time, which makes the high PDR and low energy consumed due to the shorter distance to propagate packets.
3. PDDQN-HHVBF protocol enables the pre-forwarding nodes request the Q value considering the number of candidate relay nodes. This because the more candidate relay nodes there are, the less likely the nodes are to suffer from routing holes. Thus, the method can effectively avoids the occurrence of routing holes caused by ocean current movement in sparse M-UWSNs.
4. PDDQN-HHVBF enables the pre-forwarding node to request the Q value considering the residual energy from the candidate relay nodes. This method can balance the load of the candidate relay node, thereby extending the network lifetime.
5. We proposes a \"Store-Carry-Forward\" mechanism. This mechanism store and carry packets when facing routing holes until a suitable relay node is found for forwarding, which will improve PDR and save energy of AUV significantly.  
The remainder of the paper is organized as follows: We introduce some other related work in Section 2. We introduce the concept of reinforcement learning and related algorithms in Section 3. In Section 4, we introduce the specific process of the PDDQN-HHVBF routing protocol. In Section 5, we introduce the setting of simulation parameters used in the simulation and the discussion of the simulation results. Finally, the Section 6 concludes this research and suggests some future works.