# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 3 Reinforcement Learning
Then, we calculate the gradient of the neural network weight parameter \(\omega\) relative to the loss function, and use Formula (6) to calculate the gradient value of the loss function relative to the parameter \(\omega\). \[\frac{\partial\text{L}(\omega)}{\partial\omega}=\text{E}\bigg{[}\bigg{(}\text{r }+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime}\mid\omega)-Q(\text{s},\text{a }\mid\omega)\bigg{)}\frac{\partial Q(\text{s},\text{a}\mid\omega)}{\partial \omega}\bigg{]} \tag{6}\]  
\(\frac{\partial\text{L}(\omega)}{\partial\omega}\) can be calculated from the network structure. Then, we use the gradient descent method to update the parameter \(\omega\) until the loss function converges. However, the continuous state has a strong correlation with the action input. The action in one state will generate a reward value, and updating the parameters will affect the actions in other states. So, compared with the Q-learning algorithm, DQN is unstable. This instability can greatly affect the convergence speed of DQN. Reference [31] proposed an experience replay method to solve the DQN instability problem. This method is that when the agent needs to use new data samples to update the parameters, it will also randomly sample the previously learned data to learn again, thereby avoiding the DQN instability problem caused by the correlation between state and action. The specific method is to store the state, action, reward and the next state sequence \([s,a,r,s^{\prime}]\) of the agent after each learning into an experience pool sequence \(D\). In addtion, each time the reward function is updated to reuse the training to minimize the objective function, as shown in Equation (7). \[L(\omega)=E_{(s,a,r,s^{\prime})\sim U(D)}\Big{[}\big{(}r+\gamma\max Q\big{(}s^ {\prime},a^{\prime}\mid\omega\big{)}-Q(s,a\mid\omega)\big{)}^{2}\Big{]} \tag{7}\]  
where \((s,a,r,s^{\prime})\sim U(D)\) denotes that the experience sequence \(D\) taken during the experience playback is a uniform random distribution sequence, \(s^{\prime}\) indicates the transition from the state \(s\) to the next state, and \(a^{\prime}\) indicates the action that may be performed in the environment state of \(s^{\prime}\). The above-mentioned experience replay DQN based on uniform distribution can avoid the strong correlation between data samples, which makes the training samples independent and identically distributed, and experience replay also enables reinforcement learning to learn and train in past strategies, makes the learned parameters more stable. However, this method is sensitive to the size of the experience pool. If too much experience is used, the learning time will be too long and the convergence speed will be too slow. In addition, there is a correlation between \(\gamma\max Q(s^{\prime},a^{\prime}\mid\omega)\) and the expected value of the current state \(Q(s,a\mid\omega)\) Based on the above problems, researchers proposed a method [32], which uses a new neural network to generate the target value and use it as the final policy. The idea of the way is to calculate the target value and the current state estimate separately. In this way, there is no correlation between \(\gamma\max Q(s^{\prime},a^{\prime}\mid\omega)\) and \(Q(s,a\mid\omega)\). This method is called DDQN (Double Deep Q-Network) algorithm, and the loss function of this algorithm is as Formula (8):  
\[s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},\ldots,s_{n-1},a_{n-1},r_{n},s_{n} \tag{1}\]0 \[s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},\ldots,s_{n-1},a_{n-1},r_{n},s_{n} \tag{1}\]1  
where \(\omega^{\prime}\) is the weight parameter of the target Q-Network. During the algorithm training, \(\omega^{\prime}\) will periodically synchronize from the parameter \(\omega\) of the current Q-Network. The objective value function of DDQN is shown in Formula (9):  
\[Q^{DDQN}_{\text{target}}=\bigg{\{} \begin{array}{ll}r_{i+1}&\text{$i+1$ is end}\\ \tau_{i+1}+\gamma Q^{\prime}\big{(}s_{i+1},\text{argmax}_{x}\,Q(s_{i+1},a, \omega),\omega_{i}^{\prime}\big{)}&\text{$i+1$ is not end}\end{array}