# PDDQN-HHVBF Routing Protocol Based on Empirical Priority DDQN to Improve HHVBF
## 3 Reinforcement Learning  
The PDDDQN-HHVBF protocol uses the idea of Reinforcement Learning to find the optimal relay node. In this part, we will mainly introduce the concept of reinforcement learning and related algorithms, such as Q-learning, DQN, and DDQN. Reinforcement Learning (RL) is one of the paradigms and methodologies of machine learning, which describes and solves the problem that an agent achieves specific goals in the process of interacting with the environment through learning in a way that maximizes returns. Most RL methods are based on estimated value functions, state functions or state-action pairs \((s,a)\). It assesses how well an agent is in a specific state in the environment, or how well it performs a task [3]. Usually the sample of RL is a time series, and the problem of RL is modeled and introduced into the Markov decision process (MDP). MDP is the theoretical basis for the use of RL in this paper. The schematic diagram of MDP is shown in Figure 2.  
Q-learning [3] is an important RL algorithm that is widely used in various agent systems. This algorithm has the characteristics of ensuring convergence under certain conditions and has become the most widely used RL algorithm. When using Q-learning, the agent selects a action in the current state of the environment. After the environment receives the action, it will update to a different state, and then the environment will feed back a reward signal to the agent according to the changed state. After the agent receives the reward signal, the agent chooses the next action according to the received reward signal. In general, the state parameters are random, and the action selected by the agent according to the random state is also random. Therefore, a set of data pairs of state and action, together with the rules for changing the state, constitute a Markov decision process. During this process, a sequence of states, actions and rewards \((S,A,R)\) is formed. The \((S,A,R)\) expansion is shown in Formula (1)  
\[s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},\ldots,s_{n-1},a_{n-1},r_{n},s_{n} \tag{1}\]  
where \(s_{0}\) is the initial state under environment, \(s_{i}\) refers to the state parameters of the agent in the state after the \(i\) iterations, \(a_{i}\) denotes the action that can only be selected in the \(s_{i}\), \(r_{i+1}\) is the reward value of the environment feedback after the agent performs \(a_{i}\). At the same time, the Q-learning algorithm is a Markov decision process, that is, the next state \(s_{i+1}\) in this process only depends on the current environmental state \(s_{i}\) and the selected action \(a_{i}\) in the current state. MDP will end when the state of the agent is \(s_{n}\), that is, when the agent reaches a certain state set, the MDP ends. The learning goal of the agent using Q-learning is to maximize the cumulative return in the future. Q-learning is to estimate the reward value function according to the environmental state and the behavior taken at different times. Through multiple iterations of learning, the estimated reward value function is close to the real reward value function. At the same time, the agent needs to evaluate each action and expected reward value in each environmental state. The reward value function of the action \(a_{i}\) performed by the environmental state \(s_{i}\) is \(Q(s_{i},a_{i})\), and the expected value of \(Q(s_{i},a_{i})\) is shown in the Formula (2) as shown:  
Figure 2: The schematic diagram of MDP. \[Q(s_{i},a_{i}) =\mathrm{E}\big{[}r_{i+1}+\gamma r_{i+2}+\gamma^{2}r_{i+3}+\cdots \mid s_{i},a_{i}\big{]} \tag{2}\] \[=E_{s^{\prime}}[r_{i+1}+\gamma Q(s^{\prime},a^{\prime})\mid s_{i},a _{i}]\]  
where \(r_{i+1}\) refers to the reward and return value obtained by the agent after selecting the action \(a_{i}\) from the state \(s_{i}\).