# Distributed Unmanned Aerial Vehicle Cluster Testing Method Based on Deep Reinforcement Learning
## 4 The Proposed Algorithm
### Implementation of UDTR  
This work uses intelligent experiential learning and decision strategies to design and define a deep reinforcement learning framework for virtual network mapping. Our goal is to minimize the overhead of virtual link mapping while satisfying system load balancing. The reinforcement learning system is shown in Figure 1. A deep reinforcement learning framework can be defined as follows:  
State: The state of deep reinforcement learning represents a space that accurately and objectively reflects the current environment. The mapping process can be performed according to a given directed acyclic graph (DAG). The status of UAV testing task processing node i is expressed as follows:  
\[S_{i}^{n}=\Big{\{}(lq_{i}^{n},ld_{i}^{n},ab_{i}^{n},nq_{ij}^{n})\Big{|}j\in Nb_{ i}\}. \tag{10}\]  
where \(lq_{i}^{n}\) is the link quality of the whole network where task execution node \(i\) resides, \(ld_{i}^{n}\) is the link load under the current network where node \(i\) resides, \(ab_{i}^{n}\) is the current remaining network resources available under the network where node \(i\) resides, and \(nq_{ij}^{n}\) is the queue size of adjacent node \(j\). \(Nb_{i}\) is a set of adjacent nodes of \(i\). Figure 1: The view of system architecture. Action: In this work, the action taken by node \(i\) at the current moment is recorded as \(a_{i}^{n}\left(S_{i}^{n}\right)\in A_{i}\), which is related to its current state. \(A_{i}\) is defined as  
\[A_{i}=\left\{\left.lk_{i}^{C_{h}},C_{h}\in C_{avl}\right\}.\right. \tag{11}\]  
where \(lk_{i}^{ch}\) represents \(i\) selecting the path selection scheme \(C_{h}\) to complete the entire testing task assignment. Immediate Cost: The Agent can use the current state and action to obtain a value of immediate cost from the environment as feedback. The cost function in this work is closely related to the multi-objective function, so the bandwidth resource and link energy consumption are integrated into the final cost function. \[R_{i}^{n}(S_{i}^{n},a_{i}^{n}(S_{i}^{n}))=\left\{ \begin{array}{l}k_{1}\cdot \sqrt{\frac{1}{n}\sum\limits_{i=1}^{n}(\frac{\alpha W\!L_{cpu}^{if}+\beta W\!L _{mem}^{if}}{\alpha R_{cpu}^{i}+\beta R_{mem}^{i}}}-\frac{\sum\limits_{e=1}^{n }\frac{\alpha W\!L_{cpu}^{ef}+\beta W\!L_{mem}^{ef}}{\alpha R_{cpu}^{e}+\beta R _{mem}^{e}}}{n})^{2}\\ +k_{2}\cdot\sum\limits_{l(i,i\prime)\in l_{DAG}}w_{jk}\sum\limits_{j,k\in ND }f_{j,k}^{i,i\prime}bw(i,i\prime),\text{ {task allocation succeeds;}}\\ \\ k_{fail},\qquad\text{{task allocation fails.}}\end{array} \right. \tag{12}\]  
where \(k_{1}\) and \(k_{2}\) represent weight factors. The algorithm considers both network cost and load factors, and can greatly help to minimize the learning of bandwidth cost and system load balance. As shown in Figure 2, the main process of the routing strategy design based on the DDPG is described. Generally, the overall flow of the DDPG algorithm can be divided into three main parts: sampling, training, and parameter update. In the sampling process, the algorithm randomly selects state \(s\), inputs it into the current _Actor_ network, selects action a according to state \(s\), and inputs action a into the environment, which returns a reward \(r\) and the next state \(s^{\prime}\). The quintuple (\(S\), \(A\), \(R\), \(S^{\prime}\), \(\gamma\)) is stored in the experience replay buffer for subsequent training. When the data in the experience replay buffer reach a certain amount, the algorithm enters the training process, takes out batch data from the replay buffer, and inputs them into the current _Critic_ network and _Target-Critic_ network, respectively, then calculates the loss, and updates the network parameters. Finally, the parameter update process adopts the soft update method, and the network parameters are updated every turn to ensure the stability and convergence of the algorithm. When implementing the DDPG algorithm, four networks need to be constructed first: _Actor_ network, _Target-Actor_ network, _Critic_ network, and _Target-Critic_ network.