## 4 Proposed Routing Protocol
### Routing Decision Learning Phase
#### 4.2.2 DDQN with Prioritized Experience Relay Algorithm  
In our scenario, since the action space is discrete, we adapt the value-based RL algorithms. Value-based RL is aimed at learning the optimal policy function \(\pi^{*}(a_{t}\mid a_{t})\) based on the action-value function \(Q(s_{t}\mid a_{t})\). The optimal policy function \(\pi^{*}\) is denoted as \(\pi^{*}(a_{t}\mid a_{t})=P(A=a_{t}\mid S=a_{t})\), and is used to represent the optimal probability of the node selecting the next hop \(a_{t}\) in the state \(s_{t}\). The action-value function \(Q\) is expressed as \(Q^{\pi}(a_{t}\mid a_{t})=\mathbb{E}^{\pi}[G\mid S=s_{t},A=a_{t}]\), representing the expected cumulative reward by taking the next hop \(a_{t}\) in the state \(s_{t}\) with the policy \(\pi\), where the cumulative reward \(G\) is denoted as \(G=\sum_{t=0}^{T}\gamma^{t}R_{t}\) and represents the sum of the reward values from the current node to the destination node with T hops. \(\gamma\in[0,1]\) is the discount factor, which is used to evaluate the importance of future rewards. We hope the designed routing protocol can adapt to large-scale UANETs because deep reinforcement learning is a powerful algorithm that combines the advantages of deep learning for perception and reinforcement learning for decision making, which can cause nodes to interact with the environment and intelligently make decisions in UANETs. In [42], the authors use a deep neural network to learn to approximate the action-value function, and there are two \(Q\) networks; one is the current \(Q\) network with the parameter set \(\theta\), and the other is the target \(Q^{\prime}\) network with the parameter set \(\theta^{\prime}\). The target \(Q^{\prime}\) network is always greedy in terms of selecting the maximum Q value of the next hop node as the target Q value, which accelerates the network's convergence, but it leads to overestimation and large biases. In our paper, we use double deep Q-learning (DDQN) [44] to eliminate the problem of overestimation by decoupling the action selection of the target Q value and the calculation of the target Q value. That is to say, the next state \(s^{\prime}\) is input into the current \(Q\) network to obtain the optimal Q value corresponding to the action \(a^{\prime}\). Then, the action \(s^{\prime}\) and the next state \(s^{\prime}\) are employed to obtain the target Q value in the target Q network. The update of the target Q value is as follows. \[Q_{\text{target}}=\mathbb{E}[R_{t+1}+\gamma Q^{\prime}\Bigg{(}s_{t+1},\underset {a_{t+1}}{\arg\max}Q(s_{t+1},a_{t+1},\theta_{t});\theta^{\prime}_{t}\Bigg{)} \mid(s_{t},a_{t})] \tag{11}\]  
In our work, \(\gamma\) is set as 0.9 based on our experience. The objective function of the \(Q\) network is to minimize the loss function \(L(\theta_{t})\), which is shown as:  
\[L(\theta_{t})=\mathbb{E}\Big{[}\big{(}Q_{\text{target}}\,-Q(s_{t},a_{t}; \theta_{t})\big{)}^{2}\Big{]} \tag{12}\]  
Then, the Adam gradient descent method is used to optimize the loss function. The gradient of the loss function can be depicted as \[\
abla_{\theta_{t}}L(\theta_{t})=\mathbb{E}\big{[}\big{(}Q_{\text{target}}\,-Q(s_{t },a_{t};\theta_{t})\big{)}\
abla_{\theta_{t}}Q(s_{t},a_{t};\theta_{t})\big{]} \tag{13}\]  
In [44], when calculating the target Q value, all samples \(e_{t}=(s_{t},a_{t}r_{t},s_{t+1})\) in the experience replay pool can be sampled with the same probability. However, different samples have different effects on backpropagation. The larger the TD-error, the greater the effect on the backpropagation, where TD-error \(\,=Q_{\text{target}}\,-Q(s_{t},a_{t};\theta)\). So we adapted the prioritized experience relay in [45], which produces samples where larger absolute values of TD-error are more likely to be produced as samples. Generally, the SumTree method is used to store samples with priority. The convergence rate of the current \(Q\) network is accelerated with this method.