## 3 The Proposed Method
### Training and Validation  
Algorithm 1 is the flow of the entire training architecture. The buffer capacity \(D\) is \(10^{6}\), the training step \(M\) is 40000 and every 2000 steps represents one episode. The value \(\gamma\) is 0.99 and \(K\) is 50. The value \(\lambda\) of the Q-function is 1. The programming language is python and it was written and executed on PyCharm Community Edition 2019.2.2 x64 using Xeon Silver 4110 CPU @ 2.10 GHz @ 2.10 GHz. The network layer is detailed in the simulation section. The communication rate and time delay were designed in the Q-function, in order to maximize the Q-function and obtain a better training effect, and the CBAM and VDN were added, respectively. The algorithm is based on a pure DQN without 1 and 2 modules. Any module can be embedded into the network quickly. The CBAM affects the training results along both channel and spatial aspects. With the VDN, the action selection for the UAVs is based on the global observations of all UAVs. While in the validation architecture, every transmitter of the U2U link adopts the action that can maximize the evaluation Q-value, based on the well-trained target network at every time step. Accordingly, the results of one episode were recorded and are shown in the simulation section.  
Figure 1: The overall improved DQN architecture with the CBAM and VDN.  
```
Initialize replay memory buffer to capacity \(D\). Initialize the Q-network of the UAVs with random parameters \(\theta\) and a target network of UAVs with parameters \(\hat{\theta}=\theta\). Activate environmental simulator, generating the locations of all the UAVs. Initialize the action space \(A\), including the power and frequency each UAV can choose. for step=\(1\), \(M\)do  Update positions of UAVs, channel information of U2B links \(H_{lUB}\), the U2U links \(H_{lun}\), the interference from U2U links \(I_{lun}\), the uplink reuse indicator of U2U links \(B\), the time left to transmit data \(E_{t}\) and the data left to transmit. for episode =\(1\), \(T\)do  The U2U link selects a random action with probability \(\varepsilon\), otherwise, select \(a_{t}=\max_{a_{t}\in A}Q(s_{t},a;\theta)\) with probability \(1-\varepsilon\). Implement \(a_{t}\) and move on to the next state \(s_{t+1}\) with a reward \(r_{t}\). Store the transition \((s_{t},a_{t},r_{t},s_{t+1})\) into the memory buffer. A certain amount of mini-batch transitions \((s_{t},a_{t},r_{t},s_{t+1})\) is sampled randomly from the memory buffer. Preprocess the collected state data and input the processed data into the first convolution layer of the Q-network and output the data \(M_{cnn}\). \(1\). Input data \(M_{cnn}\) into the CBAM module to obtain the data \(M_{CBAM}\). The q-values are obtained after the following convolution and relu operations are carried out on the output of the last layer. \(2\). Perform VDN operations to obtain the summation reward Q. \(\text{Set}\,y_{t}=\begin{cases}r_{j},&\text{if ends at step }t+1\\ r_{t}+\gamma\max_{a_{t}}Q(s_{t+1},a_{t};\hat{\theta}),&\text{otherwise}\end{cases}\).  Perform a gradient descent policy on the loss function \((y_{j}^{tot}-Q(\tau_{j},\mathbf{u}_{j},\mathbf{s}_{j};\theta))^{2}\) to improve the coefficients \(\theta\).  Every \(K\) step, let \(\hat{\theta}=\theta\). endfor  Save the coefficients of the target network.
```  
**Algorithm 1** The improved DQN for multi-UAV formation resource optimization with the CBAM and VDN (training stage)