# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 3 Reinforcement Learning
The learning goal of Q-learning is to find the action with the maximum reward value, that is, to maximize \(Q(s_{i},a_{i})\), the \(Q(s_{i},a_{i})\) function conforms to the Bellman formula, and the maximization of \(Q(s_{i},a_{i})\) is expressed as Formula (3):  
\[Q^{\pi}(s_{i},a_{i})=\mathrm{E}\bigg{[}r_{i}+\gamma\max_{a^{\prime}}Q^{\pi} \big{(}s^{\prime},a^{\prime}\big{)}\mid s_{i},a_{i}\bigg{]} \tag{3}\]  
where \(\gamma\max_{a^{\prime}}Q^{\pi}(s^{\prime},a^{\prime})\) is the estimated value representing the selection action of the state \(s_{i+1}\), and \(a^{\prime}\) represents the action under state \(s_{i+1}\) that the agent may perform. The \(Q\) value in the formula can also be obtained by an iterative method. In this way, the \(Q\) value stored in the agent needs to be updated in each iteration process, which will waste a lot of time and space, so this method is not adopted. When doing experimental work, we can only collect partial samples for learning. Therefore, we will use the idea of combining the Temporal Difference algorithm (TD) to calculate and update the \(Q\) value. The update method is as shown in Formula (4):  
\[Q^{\pi}(s_{i},a_{i})=Q^{\pi}(s_{i},a_{i})+\alpha\bigg{(}r_{i+1}+\gamma\max_{a ^{\prime}}Q^{\pi}\big{(}s_{i+1},a^{\prime}\big{)}-Q^{\pi}(s_{i},a_{i})\bigg{)} \tag{4}\]  
where \(\alpha\in(0,1]\), which is the learning rate of the agent,\(\gamma\) is the loss factor of the reward value, and \(a^{\prime}\) is the behavior that the agent may perform in the state \(s_{i+1}\). The formula is that the agent chooses the behavior execution that maximizes \(Q^{\pi}(s_{i+1},a^{\prime})\) among all possible actions. The agent will use a two-dimensional list to store the environmental state, the action taken and the corresponding reward value. In addtion, it queries the agent's executable action \(a_{i}\) according to the state \(s_{i}\), and obtain the corresponding reward value \(Q(s_{i},a_{i})\). The storage method of Q-learning is to store all states and corresponding actions in the agent environment, as well as the resulting reward values, in a two-dimensional form. When the environment is complex, the number of states of the agent will be very large, and there will be many actions that may be taken. At this time, the two-dimensional list of stored state-behavior reward values will be too large and complicated, not only occupying a lot of storage space of the agent, but also It will cause the agent to take too long to query the Q table, so the Q-learning algorithm is only suitable for simple environments, and the efficiency is very low in complex environments. Different from the establishment of Q table in the Q-learning algorithm, which takes up a lot of space, DQN [30] adopts the method of neural network combined with Q-learning, and obtains a function to calculate the \(Q\) value by using the parameter \(\omega\) to train and fit. Not only can it save the storage space of the agent, but also does not need to spend a lot of time to query the Q table. Only in a specific environment state, DQN can output the income value of a specific action, and then the agent only needs to select the maximum income value. The action can be executed. The learning logic of DQN is to train the neural network parameter \(\omega\) through multiple \((S,A,R)\) samples, so that the predicted return value function approximates the real return value function. The key to this approach is training the neural network. Therefore, it is necessary to determine the loss function to train the neural network. DQN uses mean squared error to define the objective function. The loss function is shown in Formula (5):  
\[L(\omega)=E\big{(}r+\gamma\max Q\big{(}s^{\prime},a^{\prime}\mid\omega\big{)} -Q(s,a\mid\omega)\big{)}^{2} \tag{5}\]  
where \(\gamma\max Q(s^{\prime},a^{\prime}\mid\omega)\) is the target value for updating the Q value in DQN, \(Q(s,a\mid\omega)\) is the current value of the Q value predicted in the neural network.