# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Bi-Directional Dynamic Interaction Process and Evolutionary Game Mechanism
As training progresses, the value loss gradually decreases and stabilizes, with D-DRL demonstrating superior convergence and estimation stability. Finally, Figure 4i further reveals the changes in the two methods, ABC-DRL and D-DRL, when dynamically adjusting the scaling factor for regulation. The regulation strategy is updated every 100 steps, and the scaling factor is usually greater than 0.5 to ensure high task completion efficiency. In some cases, the scaling factor is less than or equal to 0.5 to stimulate competition among UAVs, thereby increasing the task completion amount. In the ABC-DRL method, the duration of the scaling factor being greater than 0.5 is longer, indicating that there might be deficiencies in the task completion amount, and it needs to be optimized by reducing the scaling factor. In addition to the step-based performance comparison, we report the approximate wall time required for each algorithm to complete 12,000 episodes (1000 steps per episode) under the same hardware configuration. The estimated training durations are: D-DRL (-71 h), DRL (-67 h), and ABC-DRL (-70 h). While D-DRL incurs a slightly higher time cost, this is attributable to its hierarchical control structure and inter-agent coordination. The additional computational overhead is considered worthwhile, given the method's superior task completion performance and greater robustness in dynamic environments. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Category** & **Variable** & **Parameter Settings** \\ \hline \multirow{3}{*}{Agent Settings} & Number of convolutional layers & 2 \\  & Policy updates per horizon (agents) & 16 \\  & Agent SGD mini-batch size & 3000 \\ \hline \multirow{7}{*}{Manager Settings} & Policy updates per horizon (manager) & 4 \\  & Manager CNN dimension & 256 \\  & Manager LSTM unit size & 256 \\ \cline{1-1}  & Entropy regularization coefficient & 0.2 \\ \cline{1-1}  & Manager learning rate & 0.0001 \\ \cline{1-1}  & Manager SGD mini-batch size & 3000 \\ \cline{1-1}  & Number of convolutional layers & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Cont. Figures 5-7 show the impact of six swarm intelligence control algorithms on the following indicators during the learning evolution process of the UAV swarm control system: individual UAV task income, individual UAV task completion efficiency, the mean values of task income, energy cost, task completion amount, task completion efficiency, task throughput, and comprehensive task effectiveness. The experimental results are analyzed from individual, group, and system levels, as follows:

(1) Figure 5a-f show the distribution of individual UAV task income and task completion efficiency under different control methods. From the individual level analysis, the conclusions are as follows: (1) Under RBCA control, most UAVs have a task completion efficiency below zero, indicating intense competition within the swarm and large task income disparities; (2) Compared to RBCA, GA and ABC show an increased proportion of negative task completion efficiency, with ABC showing the largest increase. The task income disparity is reduced in both algorithms, with ABC showing the smallest difference, and task income is normally distributed; (3) Under DRL and ABC-DRL control, about half of the UAVs have a task completion efficiency greater than zero, reducing internal competition, enhancing the task completion efficiency and task income, and narrowing the

Figure 4: Training convergence curves of D-DRL, DRL, and ABC-DRL, and the trapezoidal diagram of the scaling factor for regulation. In (**a**–**h**), the x-axis represents the training steps of the D-DRL method, while in (**i**), it denotes the steps executed in the UAV swarm control simulation environment. All training curves are smoothed using an exponential moving average.