# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 4 PDDQN-HHVBF Protocol
### DDQN Algorithm Based on Empirical Priority  
This part will detail the deep reinforcement learning algorithm used in the PDDQN-HHVBF protocol. The pre-forwarding data node will select the optimal relay node according to the predicted reward value of the neural network parameters obtained from the training, specifically using the DDQN based on the priority experience value playback. The neural network used in this method mainly includes two parts: the current network and the target network. Among them, the current network is mainly responsible for the current state \(s\) of the root candidate relay node to give the action \(a\) to be executed, the network is optimized based on the policy gradient. In addtion, the target network is the evaluation of the output action \(a\) of the current network in the state \(s\). The target network is based on a value function. In the routing decision problem solved in this paper, the current network adopts a four-layer structure, and the network uses a fully connected network. The state \(s\) is used as the output layer, the second and third layers are hidden layers, and the last layer is the output of action \(a\). In this network, the hidden layer uses the ReLu activation function for nonlinearization, and the output layer uses the Tanh activation function for nonlinearization, where the output range is \((-1,1)\), and the current network of the strategy is shown in Figure 6. The target network also uses a four-layer network structure, and also uses a fully connected structure. The state-action pair \((s,a)\) is used as the input layer, the second layer and the third layer are used as the hidden layer, and the last layer outputs the evaluation value \(Q\) of the state and action pair. The target network is shown in Figure 7.  
The experience replay mechanism mentioned above can avoid the strong correlation between data samples to a certain extent, but this method is not efficient to a certain extent, because the importance of the samples is not the same, making the agent learning efficiency is not the same. Therefore, the researchers proposed a priority-based experience playback mechanism, that is, when extracting samples from the experience pool during the experience playback process, the samples are sorted and sampled according to the value of the sample priority \(rank_{i}\) from large to small. However, the samples sampled by the pool are too concentrated, which makes the error shrinking slowly and the training time is too long. Especially in the process of function estimation, the samples with high initial training errors will be frequently used in the network, which will make the learning results overfit. In response to the above problems, this paper takes advantage of the random uniform sampling of priority playback and the \(\epsilon-greedy\). The random sampling method is introduced into the priority playback mechanism so that the playback samples will not be too concentrated and lead to overfitting. The sampling probability of sample \(i\) is defined as Formula (13):  
\[P(i)=\frac{rank^{\alpha}}{\sum_{k}rank^{\alpha}} \tag{13}\]  
in the above formula, \(rank^{\alpha}\) is the priority of sample \(i\), \(\alpha\) is the priority sampling factor, when \(\alpha\) is 0, the sampling method is random uniform distribution, when \(\alpha>0\), it will not always sample with the high error data. Because the research scenario of this paper is a large-scale M-UWSNs routing problem, the required sample pool will be large. During the process of storing and sampling samples, the search and sorting of samples will make the algorithm too complex, resulting in slow convergence of the algorithm and waste a lot of time in the calculation process. Based on this problem, this paper uses the binary tree structure of SumTree to store and find samples to reduce the time of searching and sorting. The structure diagram of SumTree is shown in Figure 8.  
Figure 6: The schematic diagram of the current network. Figure 7: The schematic diagram of the target network.