## 3 The Proposed Method
### Introduction of the DQN  
The multi-UAV movement problem could be modeled as the constrained Markov decision process (CMDP) problem, giving a moving area that is classical in order to solve RL tasks with constraints. The maximum cumulative long-term discounted reward can be obtained after the agents are well trained to execute the best action strategy. In the multi-UAV formation network, the whole network corresponds to the environment in the RL method, and each U2U link is regarded as an agent. An action is labelled with \(a_{t}\) and it includes two parameters of the action space \(A\). Each U2U transmitter makes an action choice according to the current state \(s_{t}\), and the environment feedbacks a reward \(r_{t}\) and the next state \(s_{t+1}\). Therefore, state, action, reward and the next state \([s_{t},a_{t},r_{t},s_{t+1}]\) are the main transitions in RL. The value \(A\) is a set of \(M\) uplinks and three power levels. The state \(s_{t}\) contains the channel information in the U2B links \(H_{UB}\) and U2U links \(H_{uu}\), the interference from the U2U links \(I_{uu}\), the uplink reuse indicator of the U2U links \(B\), the time left to transmit data \(E_{t}\) and the data left to transmit \(D_{t}\). The set of all states is called the state space \(S\).  
For a given state \(s_{t}\), in each time slot the U2U link transmitter needs to select an action according to the strategy \(\pi\). In regular Q-learning, the decision-making strategy is carried out by a q-function. Once a state-action pair is determined, the q-value reward is calculated after selecting one action. To find the optimization strategy to maximize the cumulative reward, once the q-value is calculated, the action selection strategy can be carried out according to the following expression  
\[a_{t}=max_{a\in A}Q(s_{t},a). \tag{16}\]  
So, the optimization strategy can be denoted as \(\pi:s_{t}\in S\to a_{t}\in A\).  
For a specific action and state, the maximum Q-value, i.e, the expected accumulative discount reward, is expressed as  
\[Q(s,a)=E[\sum_{i=0}^{\infty}\lambda^{i}r_{t+i}], \tag{17}\]  
where \(\lambda\) is the discount coefficient that is needed to strike a balance between current and future decisions. Equation (17) is the total cost for the optimization. According to Equation (15), each UAV can choose the most appropriate power and spectrum based on its remaining data transmission time and the system interference.  
In the multi-UAV formation problem, the state and action space will be very large, which makes it is difficult to use regular Q-learning as some Q-values are seldom updated. As a combination of RL and a deep network, the DQN has been widely used. A DQN can take a state as the input and output all the action values \(Q(s,a;\theta)\). It can then select the action with the maximum value as the next action. The values of \(\theta\) are the parameters of the neural network (NN) and are called the Q-network. As is well-known, in the study of [22], [PERSON] et al. introduced target networks and experience replay to achieve a better performance for the DQN algorithm. The output of target network in the DQN is written as  
\[y_{t}^{DQN} =r_{t+1}+\gamma max_{a}Q(s_{t+1},a;\theta) \tag{18}\] \[=r_{t+1}+\gamma max_{a}E[\sum_{i=1}^{\infty}\lambda^{i}(R_{t+i}+ T_{t+i})],\]where \(\hat{\theta}\) are the coefficients of the target network and are updated according to the Q-network after certain steps. The coefficients of the Q-network are updated according to the gradient descent policy to loss function. The loss function is shown as below.  
\[Loss(\theta)=\sum_{(s_{t},a_{t}\in D)}(y-Q(s_{t},a_{t};\theta))^{2} \tag{19}\]  
where \(D\) is a memory buffer that is used to store all the transitions \([s_{t},a_{t},r_{t},s_{t+1}]\). Referring to the experience replay method, in order to reduce the data correlation, fixed batch transitions are randomly selected from D in every training period. As the input of the Q-network, parameters of the Q-network are updated while training the data.