# Distributed Unmanned Aerial Vehicle Cluster Testing Method Based on Deep Reinforcement Learning
## 4 The Proposed Algorithm
### Implementation of UDTR
The current _Actor_ network inputs the current state, and outputs the current action. The _Target-Actor_ network inputs the next state and outputs the next action. The current _Critic_ network inputs the current status and current action, and outputs the current \(Q\) value. The _Target-Critic_ network inputs the next state and next action, and outputs the next \(Q\) value. These four networks work together to enable the algorithm to learn the optimal policy. In this work, we establish the target network and obtain the target value of training the _Critic_ network. The target network includes the _Target-Actor_ network and the _Target-Critic_ network. The input of the target network can be sent to the experience replay buffer as the next state (i.e., \(S_{t+1}\)). A critical value can be obtained as the output of the target network for _Critic_ training. The experience replay buffer is used to store experience tuples, including the current state, selection actions for task assignment decisions, immediate cost, and next state. A well-designed experience tuple can be randomly sampled for the training of the main network and the target network, which helps to reduce the influence of data correlation. In Figure 2, we can also see that the agent in the UTDR method proposed in this paper can explore the environment according to the current network conditions and design the corresponding path selection decision actions. The path selection scheme is then sent to the selected available nodes to achieve connectivity across the network. Using the path selection scheme in action, the UAV nodes can realize network resource planning after the task assignment process and cooperate with the subsequent link scheme selection. Finally, the system feedback is used to calculate the cost of the distributed testing task assignment strategy adopted. The procedure is as follows:  
**Step 1:** Pass the state of the network environment to the primary network _Actor_ of the main network and perform replay buffering. In general, during a training cycle, transition information for each interaction with the environment is stored in the experience replay buffer. At the same time, the learning batch of the neural network consists of the transition process information sampled from the experience replay buffer. **Step 2:** Based on the current state and experience tuple, the agent uses the deep model of the main network and the target network to make a decision on the next action. The action of the agent is to select the path set of the calculation result transmission in order based on the given DAG. The _Target-Actor_ and _Target-Critic_ in the target network are constructed, and the _Target_ policy value \(\theta_{\mu^{\prime}}\) and the target \(Q\) value \(\theta_{Q^{\prime}}\) are updated according to the empirical tuple, respectively. Therefore, by taking the _Target-Critic_ of the _target_ network, we can calculate that the target \(Q\) is  
\[\textit{Target\_Q}=r_{t}+\gamma Q^{\prime}(s_{t+1},\mu(s_{t+1}|\theta_{\mu}) \theta_{\mu\prime}). \tag{13}\]  
We can then set the result to the critical value of the primary network. In order to update the _Critic_ module in the deep network, the _Critic_ network realizes the update of the primary \(Q\) value \(\theta_{Q}\) by overcoming the optimization problem of minimizing the Loss function \(Loss(\theta_{Q})\) after receiving _Target-Q_. Using the tuple \(\theta_{Q}\) in the experiential replay buffer, the result of the policy gradient is computed by the _Actor_ of the main network to calculate the next action. **Step 3:** The UTDR agent uses the determined actions to try out the available policy on each communication network path of the UAV cluster. After execution path selection, each state of the agent is passed from one state to another. The agent then receives feedback from the environment with immediate system costs based on the new state. When the current link scheme fails to be established, the agent receives the cost return value \(k_{\textit{fail}}\).