# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Bi-Directional Dynamic Interaction Process and Evolutionary Game Mechanism
To analyze the performance differences of different swarm control algorithms, this paper designs a comparative experiment, where the UAV's performance, task initialization conditions, and total funding are identical to ensure a fair comparison. To quantify the performance of the algorithms in terms of resource allocation and reward balance, the UAV reward function is formulated as follows:

\[Reward_{t,i}=\frac{\beta*Revenue_{t}}{N}+(1-\beta)*Revenue_{t}*\,\,Proportion_{t,i} \tag{11}\]

where \(Proportion_{t,i}\) represents the fraction of tasks completed by UAV \(i\) relative to the total tasks in the swarm at time \(t\). \(N\) denotes the total number of UAVs. \(Revenue_{t}\) represents total funds within the period. \(\beta\) is the ratio of basic income to performance incentive, dynamically adjusted according to the control strategy. In this context, the agent's task income can be defined as the benefit in each cycle, determined by the fraction of tasks completed by the UAV relative to the total tasks in the swarm and the manager's control strategy. Furthermore, this paper evaluates the algorithms from three levels: individual, group, and system. At the individual level, the focus is on task income and task completion efficiency. At the group level, trajectory heatmaps are analyzed to examine agent interactions. At the system level, the mean values of task income, energy cost, task completion amount, task completion efficiency, task throughput, and comprehensive task effectiveness are measured [41]. In this study, we compare five representative and advanced UAV swarm control algorithms, as summarized in Table 3. The compared algorithms are as follows: the centralized rule-based control architecture method proposed by [PERSON] et al. [15] (referred to as RBCA), the distributed random algorithm based on an improved genetic algorithm proposed by [PERSON] et al. [19] (referred to as GA), the distributed artificial bee colony path planning method based on multi-strategy evolutionary learning proposed by [PERSON] et al. [20] (referred to as ABC), the hybrid distributed collaborative search method based on multi-agent reinforcement learning proposed by [PERSON] et al. [25] (referred to as DRL), and the method combining artificial bee colony algorithm with single-layer reinforcement learning (referred to as ABC-DRL). The inclusion of ABC-DRL as a comparative method aims to demonstrate the necessity of adopting a double-layer architecture with deep reinforcement learning. Given that DRL outperforms ABC in handling function fitting [26], ABC-DRL adopts the previously mentioned distributed ABC method as the inner model of the double-layer architecture, while the outer model employs a DRL structure consistent with D-DRL. Table 3 clearly presents a side-by-side comparison of these methods in terms of the design, operational characteristics, and performance metrics. The optimization goals of RBCA, GA, ABC, and DRL are to improve comprehensive task efficiency, while ABC-DRL and D-DRL perform multi-level optimizations by optimizing the task completion amount in the inner layer and comprehensive task effectiveness in the outer layer. Under the condition that all reinforcement learning methods converge, the superiority of the double-layer deep reinforcement learning method is verified by comparing the performance metrics of different algorithms. The experimental environment uses a 25 \(\times\) 25 grid, where the state is represented as a three-dimensional tensor of size 25 \(\times\) 25 \(\times\) c, with c indicating the number of agents in each cell. The system parameters in this study are based on a UAV swarm produced by a collaborating helicopter company and are mapped to real-world application scenarios. To protect the company's interests, certain parameters were adapted from established literature sources [27; 28; 29; 30].