# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 2 Related Works  
In this part, we mainly review the related routing protocols of UWSNs and analyze the advantages and disadvantages of these protocols. There are some location-based routing protocols, such as VBF (Vector-Based Forwarding) [12], HHVBF (Hop-by-Hop Vector-Based Forwarding) [13] and AHHVBF (Adaptive Hop-by-Hop Vector-Based Forwarding) [14]. VBF utilizes location information to improve the energy efficiency of dynamic networks. In VBF, data packets are delivered within the range of the routing pipeline, and nodes within this range will be used more frequently than nodes outside this range, which will lead to unbalanced energy consumption and shorten network life. In order to solve the problem of VBF, HHVBF, and AHHVBF are proposed,respectively. The HH-VBF protocol and the AHHVBF protocol also use the idea of routing pipeline. Unlike VBF, both HHVBF, and AHHVBF do not use a fixed pipeline to transmit data, but establish a new pipeline after each data transmission. In this way, compared with VBF, HHVBF, and AHHVBF protocols can still transmit data efficiently in areas with low node density, so they have higher packet delivery ratio, and each node can be used evenly, resulting in longer network lifetime. However, using the two protocol, a large network with a large number of nodes has a high data collision rate and low network lifetime. There are some depth-based routing protocols. In [15], [PERSON] et al. proposed a depth-based routing (DBR) protocol, which effectively reduces the energy consumption and the number of collisions. The DBR protocol uses a greedy algorithm to delivery data packets from the sender to the receiver according to the depth information. However, if there is no surface receiver near the node, the data packets carried by the node will be discarded. In addition, the data packets can only be bottom-up transmission cannot meet the requirements of surface receivers to transmit data down. Based on the DBR protocol, the fuzzy depth-based routing protocol (FDBR) [16] comprehensively considers hop count, depth and energy information to select forwarding nodes, thereby improving the energy efficiency and end-to-end delay performance of the UWSNs. High latency is also a challenge for underwater acoustic communication due to the slow propagation speed of acoustic signals in UWSNs. [PERSON] et al. [17] proposed an EE-DBR protocol that no longer uses the depth threshold used in DBR to narrow down the set of candidate nodes, instead uses ToA (Time of Arrival) technology [18] to remove a fixed area to narrow down the range of the candidate forwarding set. However, when nodes are deployed sparsely, the algorithm cannot effectively solve the routing hole problem. There are other depth-based routing protocols, such as VAPR (Void-Aware Pressure Routing) [19], IVAR (Inherently Void Avoidance Routing) [20], and OVAR (Opportunistic Void Avoidance Routing) [21], etc. These methods make use of hops, distance, forwarding direction and other information to keep packets away from the routing void during forwarding. However, on the premise of guaranteeing data transfer rate, these methods sacrifice more real-time performance, and the end-to-end delay is high. At the same time, the state change of some nodes will also lead to the state change of many other nodes, resulting in higher network overhead. Although traditional routing protocols have improved network performance, the underwater environment is relatively harsh, and routing protocols must still deal with many constraints [22]. Today, Multi-Agent Reinforcement Learning (MARL) [23] has been successfully applied in multiple domains involving distributed decision making. A fully distributed underwater routing problem can be thought of as a cooperative multi-agent system. In [24], adaptive, energy-efficient, and lifetime-aware routing protocol based on Q-learning (QELAR) uses the method of Q-learning to calculate its own routing decision.