# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 2 Related Works
QELAR comprehensively considers the energy consumption of sensor nodes and the remaining energy distribution between adjacent nodes when designing the Q-value function, thereby optimizing the total energy consumption and network lifetime. However, the simulation model adopted by QELAR is for fixed nodes, which is not suitable for the situation where the movement of ocean currents causes node drift changes. In order to improve the real-time performance of the algorithm, an underwater multistage routing protocol (MURAO) based on Q-learning is proposed in [25]. MURAO uses clustering to divide sensor nodes into algorithmic node sets for multiple sensors. The nodes in the cluster aggregate the collected data to the cluster head, and the cluster head routes to the sink node through multiple hops. MURAO allows multiple clusters to route data to sink nodes in parallel, improving real-time performance, but increases redundant data in the network and may cause data conflicts. In [3], a DQN-based routing decision protocol DQELR with adaptive energy consumption and delay is proposed, which effectively prolongs the network lifetime of UWSNs. In addtion, the protocol adaptively selects the optimal node as a relay node according to the energy and depth state of the communication phase. However, this protocol uses DQN with a slow convergence speed, and does not consider the routing hole problem in the sparse network state, and is applicable to a single scenario. Ref. [26] proposed a reinforcement-learning based routing congestion avoidance (RCAR) protocol. In order to improve energy efficiency and reduce end-to-end latency, the RCAR makes routing decisions based on congestion and energy. However, they do not consider void holes and cannot guarantee the reliability of transmission. Moreover, [PERSON] et al. [27] designed a routing protocol for UWSNs, which focus on the characteristics of high energy consumption and high latency. This paper investigates a Q-learning-based localization-free routing protocol (QLFR) to prolong the lifetime as well as reduce the end-to-end delay for underwater sensor networks. However, the protocol does not consider the problem that nodes with large Q values become routing holes due to the influence of move current movement. [PERSON] et al. [28] proposed a Q-learning based multi-hop cooperative routing protocol for UWSNs named QMCR, which can automatically choose nodes with the maximum Q-value as forwarders based on distance information. Aiming to seek optimal routing policies, Q-value is calculated by jointly considering residual energy and depth information of sensor nodes throughout the routing process. In addition, they define two cost functions (depth-related cost and energy-related cost) for Q-learning, in order to reduce delay and extend the network lifetime. The algorithm considers the overall energy consumption of the system, but does not consider the individual energy consumption of nodes. Therefore, the problem of short network life due to common relay node energy depletion will occur. Aiming at the problem of routing holes, [PERSON] et al. [1] proposed a routing algorithm RLOR based on Q-learning, which combines the advantages of opportunism routing and reinforcement learning algorithms. The RLOR is a kind of distributed routing approach, which comprehensively considers nodes' peripheral status to select the appropriate relay nodes. Additionally, a recovery mechanism is employed in RLOR to enable the packets to bypass the void area efficiently and continue to forward, which improves the delivery rate of data in some sparse networks. However, this protocol only has good performance for sparse networks. For large-scale UWSNs, this protocol does not effectively reduce data conflicts. For large-scale UWSNs, this protocol throughput and energy consumption performance are not good. [PERSON] et al. [29] developed a reinforcement learning-based opportunistic routing protocol to enhance transmission reliability and reduce energy consumption.