# Multi-Objective Optimized GPSR Intelligent Routing Protocol for UAV Clusters
## 4 DDQN-MTGPSR Protocol Design
### DDQN Network Construction  
Figure 2 demonstrates the application of the DDQN model to the UAV cluster routing decision problem, reflecting the interaction between the model and the network environment. The agent uses the current network state \(s_{t}\) and selects the next hop routing node based on the policy \(\pi\) to select the next-hop routing node, and the network state is further transformed to \(s_{t+1}\) after the environment performs the action and gives a reward \(r_{t}\).  
If there is a demand for packet forwarding in the current node, it needs to get the state information of each neighbor as shown in (5) according to the destination node information stored in the packet and the routing table of this node, and integrated into the state space \(s_{t}\). Each network of DDQN consists of input layer \(IN\), hidden layer \(Hd\), and output layer \(L_{o}\). The hidden layer consists of a convolutional layer \(Conv\) and two fully connected layers \(Fc_{1}\) and \(Fc_{2}\). If the total number of neighbor nodes is \(N_{nb}\), then the input data is a two-dimensional matrix of size \(N_{nb}\times 7\), where \(7\) is the number of state eigenvalues.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Field 1** & **Field 2** & **Field 3** & **Field 4** & **Field 5** & **Field 6** & **Field 7** & **Field 8** \\ \hline \(ID_{1}\) & coordinate & moving & delay & energy & \(NBT_{1}\) & \(SNR_{1,i}\) & timestamp \\ \(ID_{2}\) & (geometry) & model & delay & energy & \(NBT_{2}\) & \(SNR_{2,i}\) & timestamp \\..... &..... &..... &..... &..... &..... &..... &..... &..... \\ \(ID_{n}\) & coordinate & moving & delay & energy & \(NBT_{n}\) & \(SNR_{n,i}\) & timestamp \\ \((geometry)\) & model & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Improved GPSR routing table RT.  
Convolutional layer for hidden layer \(Conv\) : The size of the convolutional kernel is \(2\times 2\), stride is 1, and no padding in both height and width directions. The convolutional layer has a total of \(N_{hr}=(N_{nb}-1)\times\left(n_{FeatureMap}-1\right)\times n_{FeatureMap}\) neurons, where \(n_{FeatureMap}\) denotes the number of feature maps (equals 7). Fully connected layers \(Fc_{1}\) and \(Fc_{2}\): the number of neurons of \(Fc_{1}\) can be taken in the interval range of \([4N_{nb},6N_{nb}]\), the number of neurons of \(Fc_{2}\) can take the value of \(N_{nb}\). Each fully connected layer uses a rectified linear unit (ReLU).  
Input \(s_{t}\) into the DDQN model and complete the mapping from \(N_{nb}\times 7\) dimensional environmental data to one-dimensional output data, and the final output layer \(L_{o}\) outputs the Q-values corresponding to all \(N_{nb}\) nodes. Further, based on the Q-values corresponding to the actions and based on the policy \(\pi\) to decide the next hop routing node. In order to balance the relationship between exploration and utilization as much as possible, we adopt the \(e-greedy\) strategy, in which the agent has the probability of \(\varepsilon(e<1)\) choosing a random action with unknown reward and the probability of \((1-\varepsilon)\) choosing the action with the highest value among the existing actions with when making decisions.