# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
###### Abstract  
Reinforcement learning (RL) has been successfully applied to underwater routing protocols due to its powerful ability of distributed decision making. However, the traditional RL has slow convergence speed and low learning efficiency in underwater. Meanwhile, too many studies focus on using RL to find low hop paths rather than short distance paths in underwater routing, while the long distance of ocean communication is the significant reason for the packets collision and energy loss in underwater. Based on the above problems, this paper proposes the PDDQN-HHVBF (Empirical Priority DDQN to Improve Hop-by-Hop Vector-Based Forwarding) protocol for M-UWSNs (Mobile source node Underwater Wireless Sensor Networks), in which AUV (Autonomous Underwater Vehicle) is used as source node to collect data and transmit data hop-by-hop to Sink node through underwater nodes. The proposed protocol is adopt to find the optimal relay nodes in pipeline referred HHVBF protocol by requesting the max Q value according to three states of the residual energy of nodes, the number of candidate relay nodes and the geographical location information of all candidate relay nodes in time. This because PDDQN-HHVBF avoids the strong correlation between data samples, and its playback samples will not be too concentrated or lead to over fitting. It can converge rapidly in underwater environment. In addition, the requesting Q value mechanism related to the geographical location information can find the optimal relay node with short distance propagation in large-scale networks, which will reduce the number of packets collision, and then saving energy and improving network lifetime. In addition, the in-time requesting for Q value can cope with the nodes drift affected by ocean current movement. In addition, the Q value related to the residual energy of nodes and the number of candidate relay nodes will effectively load balancing nodes, prolong network lifetime and avoid routing holes. Finally, the \"Store-Carry-Forward\" mechanism proposed for AUV, this mechanism store and carry packets when facing routing holes until find the optimal relay node for forwarding, which will improve PDR and save energy of AUV significantly. The simulation results show that, the proposed PDDQN-HHVBF protocol converges about 30% faster than DQELR. Although its delay is higher than DQELR and ROEVA for requesting Q value. It outperforms VBF, HHVBF, DQELR, and ROEVA in terms of energy efficiency, PDR, and lifetime, which are analyzed by varying speed of nodes from 0 m/s to 3 m/s with 1000 nodes and varying number of nodes from 500 to 3000 with speed in 1 m/s. M-UWSNs; reinforcement learning; AUV; PDR; lifetime; routing hole +
Footnote †: journal: _electronics_  
geographic and meteorological data collection, earthquake/tsunami disaster prevention, water pollution monitoring, and national defense security. It has received increasing attention in the field of marine science and technology. In addtion, it has become an indispensable tool for human exploration of the marine world [1; 2; 3; 4]. In addition, sixth generation (6G) systems enable users to share information globally across a broad spectral range, and in full application with extremely low latency and high data rates. In recent years, the demand for mobile communication Stronger capabilities, wider scope, and growing in the industrial field [5; 6; 7; 8]. Similarly, the application scenarios of 6G in underwater are also extremely wide, and it is also a direction that researchers need to pay attention to in the future. UWSNs is composed of sensor nodes deployed underwater, the Sink node on the sea surface, and autonomous underwater vehicle (AUV), which use acoustic waves to transmit information to each other [9]. Among them, the sensor node can be used as a collection node to collect ocean data and forward it to the relay node, or it can be used as a relay node to receive the data collected by the collection node and then forward it to the Sink node hop by hop.