## 4 Dueling DQN-Based Routing Protocol
### The Proposed Routing Protocol
#### 4.3.2 Routing Decisions  
After the neighboring table was established, when a source node sent a data packet to the destination node, the routing decision process based on the dueling DQN algorithm was triggered. With the assistance of relay-node forwarding, the data packets finally reached the destination node.  
When current forwarding node \(i\) needed to forward a packet, the process of the routing decisions was the following. First, current forwarding node \(i\) had to determine the number of its optional actions \(A_{i}^{\text{opt}}=A_{i}-HVN\). If \(A_{i}^{\text{opt}}=\emptyset\), current node \(i\) discarded the packet. If \(A_{i}^{\text{opt}}\
eq\emptyset\), it proceeded to the next step. Second, current forwarding node \(i\) extracted the feature information \(F_{x}=\{l_{i},l_{i},d_{x},l_{x},Q_{i,x},T_{x}^{\text{wait}},NF_{x}\}\) from the neighboring table and received the data packet (where \(l_{i}\) is the location of current forwarding node \(i\), \(l_{i}\) is the location of last-hop forwarding node \(l\), \(l_{d}\) is the location of destination node \(d\), \(l_{x}\) is the location of neighboring node \(x\), \(LQ_{i,x}\) is the link quality of link \(L_{i,x}\), \(T_{x}^{wait}\) is the total waiting time of packets in the buffer of neighboring node \(x\), and \(NF_{x}\) is the neighboring topology information). According to these features, the state information of node \(i\) was \(s_{i}=\left\{d_{i,x},d_{x,d},d_{i\overline{u}_{2},d},d_{\text{sum}}\,LQ_{i,x},DIR_{i,d},T_{x}^{\text{wait}}\right\}\). For a detailed definition of \(s_{i}\), please refer to Section 4.1 (1). Third, we fed the normalized state \(s_{i}\) into the dueling DQN network. Finally, current node \(i\) selected the next-hop node according to the Q-value output by the Q-network. If the Q-network was in the training phase, the strategy of the next-hop node selected depended on expression (19). The optimal decision was learned through iterative learning until the Q-network converged. In the testing phase, each node selected the next-hop node depending on the softmax function (20). The pseudocode of the routing decision process is presented in Algorithm 1.  
Figure 4: The format of the packets. (**a**) Hello packet. (**b**) ACK packet. (**c**) Data packet.  
```
0:\(L_{i}\): is the location of the current forwarding node \(i\); \(p\): is the data packet to be forwarded; \(NT_{i}\): is the neighboring topology information;
0: Select the next-hop node according to the Q-value;
1: Obtain a set of neighboring nodes \(A_{i}\) by \(NT_{i}\);
2: Calculate the optional neighboring set \(A_{i}^{\text{opt}}\) that excludes nodes that the packet has visited (HVN)
3:if\(A_{i}^{\text{opt}}\)=\(\emptyset\)then
4:\(a^{*}=-1\) and packet loss
5:else
6:for\(x\) in \(A_{i}^{\text{opt}}\)do
7: Extract feature information \(F_{x}=\left\{l_{i},l_{l},l_{x},LQ_{i,x},T_{x}^{\text{wait}},NF_{x}\right\}\) and calculate state feature \(s_{i}=\left\{d_{i,x},d_{\text{x,d}},d_{\text{x,d}},d_{\text{sum}},LQ_{i,x},DIR_{ i,d},T_{x}^{\text{wait}}\right\}\)
8: Input \(s_{i}\) into our RL model and output \(Q_{i}\)
9:if the Q-network is in training phase then
10: Select next step according to (19);
11: Store \(e_{t}=(s_{t},a_{t},r_{t},s_{t+1})\) into experience set \(d_{t}=\left\{e_{1},\cdots,e_{t}\right\}\);
12: Update Q-network parameters \(\mathbf{\omega},\mathbf{\alpha},\mathbf{\beta}\) and \(\mathbf{\omega}^{\prime},\mathbf{\alpha}^{\prime},\mathbf{\beta}^{\prime}\)
13:else
14: State select next step using (20)
15:endif
16:endfor
17:endif
```  
**Algorithm 1** Delay-aware and link-quality-aware geographical routing forwarding protocol for UANET via dueling deep Q-network at current forwarding node \(i\)