# Multi-Objective Optimized GPSR Intelligent Routing Protocol for UAV Clusters
## 3 Mathematical Modeling of Multi-Objective Routing Optimization Problems
### Deep Double Q-Learning Network  
The sequential behavioral decision problem of the DDQN algorithm is defined by the MDP, which introduces an Agent that learns a value function, which is associated with Bellman's equation, and together, guide the agent's decision-making process. In this study, we use this model to model the routing process of UAV clusters. MDP consists of the tuple \(\{S,A,P,R\}\), where S, A, P, and R denote the state space, action space, state transfer probability, and the reward function, respectively. The agent takes an action \(a\) at \(s\) according to the policy \(\pi\). The environment sends a reward \(r\) to the agent based on the agent's action \(a\) and transfers to the next state \(s^{\prime}\). The agent acquires the experience tuple \(e=(s,a,r,s^{\prime})\). The goal of the intelligence is to learn the optimal strategy to maximize the desired reward. The DDQN algorithm is a variant of Q-learning, a non-strategic RL algorithm based on model-free values, based on the action value function \(Q(s,a)\), iteratively updating the Q-value function using [PERSON]'s equation and gradually approximating the optimal value function to obtain higher rewards. The Bellman's optimal equation for QL is:  
\[Q^{*}(s,a)=r(s,a)+\gamma\sum\
olimits_{s^{\prime}\in S}P(s^{\prime}|s,a)\max Q ^{*}_{a\in A}\big{(}s^{\prime},a^{\prime}\big{)} \tag{1}\]  
where \((s,a)\) denotes the reward value for taking action a under s, \(\gamma\) denotes the discount factor, and \(P(s^{\prime}|s,a)\) denotes the execution of action \(a\) the probability of transferring from states to s' afterward. \(Q^{*}(s,a)\) is approximated by the following equation:  
\[Q(s,a)\gets Q(s,a)+\alpha\big{[}r+\gamma Q\big{(}s^{\prime},a^{\prime} \big{)}-Q(s,a)\big{]} \tag{2}\]  
where \(\alpha\) represents the learning rate. The optimal policy \(\pi^{*}\) is the policy that makes \(Q(s,a)\) approach \(Q^{*}(s,a)\). The DDQN uses a deep neural network to approximate \(Q(s,a)\), which gradually learns an accurate action value function to select the optimal action by introducing a deep neural network, defining an empirical playback region, computing the target value, and updating the network parameters in a gradient descent inverse manner. (1) Experience playback: the experience playback mechanism works by storing experiences in order to compose a sequence of experiences \(D=\{e_{1},e_{2},...e_{n}\}\). During training, experience samples are randomly selected from D randomly selected experience samples and adopts the gradient descent algorithm when updating network parameters, which improves the efficiency of utilizing historical data and reduces data correlation. (2) Target network \(w^{\prime}\): \(DDQN\) utilizes a deep neural network to approximate the action value function, which is denoted as \(Q(s,a;\theta_{i})\), where \(\theta_{i}\) denotes the parameters of the network at the i-th iteration. The optimization objective value is \(y^{DDQN}=r+\gamma Q\bigg{(}s^{\prime},\underset{a}{\text{argmax}}Q(s^{\prime},a;\theta_{i});\theta_{i}^{-}\bigg{)}\), the objective value is calculated by the target network \(w^{\prime}\) and evaluation network \(w\), \(s^{\prime}\) is the next state, \(a^{\prime}\) denotes the possible actions, and \(\theta_{i}^{-}\) is the parameter of the target network.