# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Bi-Directional Dynamic Interaction Process and Evolutionary Game Mechanism
The learning process is structured as follows:

(1) Pre-training in a no-manager control environment: Agents undergo an initial training phase in a flexible setting, allowing them to adapt to diverse strategies before the introduction of control constraints. (2) Control strategy integration and entropy regularization: A control strategy is introduced, and entropy regularization is applied to promote strategy diversity and prevent premature convergence, thereby stabilizing the optimization process. (3) Gradual integration of the manager's control strategy using simulated annealing: To prevent unstable learning dynamics caused by the sudden introduction of the manager's control strategy, a simulated annealing algorithm during the early phase of training is employed. Specifically, for the first one-quarter of the training duration, the influence of intervention strategies is reduced to zero. This gradual introduction ensures a smoother adaptation process, mitigating abrupt disruptions in agent learning. Entropy regularization is achieved by incorporating a weighted entropy term into the strategy gradient objectives of both the agents and the manager, as shown in the following formula:

\[\text{Entropy}(\pi)=-\text{E}_{a\sim\pi(.|s)}[log\ \pi(a|s)] \tag{8}\]

where entropy regularization enhances strategy exploration, stabilizes the solving process, and prevents training from failing to converge due to strategy mutations. ## 4 A Case Study of Intelligence Control of Unmanned Aerial Vehicle Swarm

### Design of Experimental System Scenarios

This paper constructs a UAV swarm control system to validate the effectiveness of the swarm intelligence control method based on double-layer deep reinforcement learning. The UAV swarm is chosen as the research subject not only because it shares similar operating mechanisms with other swarm control systems, but also due to its highly complex dynamic interaction characteristics. In practical applications, UAV swarms must operate in complex environments while addressing multi-dimensional tasks and managing dynamic agent collaboration [39]. Such operational complexities demand advanced intelligent decision-making capabilities form the control system. By analyzing the evolutionary process of a UAV swarm control system through simulations, conclusions are derived that are applicable not only to UAV swarms but also potentially extendable to other swarm intelligence systems. This broad applicability underscores the universality and reliability of the research results. In complex and dynamically changing environments, UAV swarms performing strike missions often struggle to distinguish between high-value and low-value targets. The difficulty stems from technical limitations, such as endurance constraints, limited storage capacity, restricted communication ability, and the algorithmic complexity in target recognition system. The assessment of high-value targets requires more energy, which in turn impacts task prioritization and the optimization of energy costs [40]. The manager can adjust strategies through data analysis and incentive mechanisms to increase the task completion amount. However, the overwhelming number of low-value targets compared to high-value ones leads UAVs to prioritize attacking low-value targets to maintain a competitive advantage. This results in challenges in simultaneously optimizing both task completion amount and task completion efficiency. The UAV swarm intelligence control system developed in this paper includes three roles: UAVs, a manager, and tasks, with tasks allocated to target areas based on predefined rules, as shown in Figure 3. The system is divided into three task areas: i, ii, and iii (low-, medium-, and high-value task areas). The UAVs autonomously explore targetswithin their field of view and strike them, while dynamically increasing energy costs. A task is regarded as successfully completed when at least one UAV reaches the task location (i.e., the corresponding grid cell) before the task's availability window expires.