# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Bi-Directional Dynamic Interaction Process and Evolutionary Game Mechanism
During this period, no other UAVs are permitted to interfere with or compete for the task. Once these conditions are satisfied, the task is removed from the environment and a corresponding reward is assigned to the agent responsible. This definition ensures that task completion is constrained both spatially and temporally, and is directly linked to each agent's ability to manage limited energy and make strategic decisions in a decentralized multi-agent setting. Additionally, new tasks are automatically generated during processing. The manager selects the control strategy based on the UAV's performance and allocates resources accordingly. Various experimental scenarios are created through different control strategies and parameters to analyze the evolution patterns of the UAV swarm control system and the optimization of strategies. In the experimental system, UAVs make decisions based on environmental changes, task locations, control strategies, and the behavior of neighboring UAVs within their field of view. They choose a direction to move (up, down, left, or right) or remain stationary. Tasks are randomly generated, and incomplete tasks may disappear, with new tasks continuously being generated. UAVs with varying performance levels yield different rewards and consume varying amounts of energy when completing the same task (high-performance UAVs provide higher rewards and incur lower energy costs). The state space consists of multiple feature components, including: Environmental dynamics (task locations, task rewards, and energy requirements within the UAV's field of view), Agent-level context (the UAV's own energy level and performance capability), Neighboring agents (behavior and status of other UAVs in nearby grid cells), and High-level policy input (a control signal from the manager representing the global strategic directive). The UAVs' strategy network outputs decisions in the form of a probability distribution, as shown in the following formula:

\[a_{i,t}\sim\pi_{a}\Big{(}o_{i,t}^{\text{vision}},o_{i,t}^{\text{outer}},o_{i,t }^{\text{inner}},o_{i,t}^{\text{policy}},h_{i,t-1};\theta\Big{)} \tag{9}\]

where the strategy network simultaneously updates the hidden state, \(h_{i,t-1}\), with inputs, including spatial observation values, \(o_{i,t}^{\text{vision}}\), within the UAV's field of view; the rewards and task information of UAVs in the field of view, \(o_{i,t}^{outer}\) ; the energy cost and capabilities of UAVs in the field of view, \(o_{i,t}^{\text{inner}}\); and the manager's control strategy, \(o_{i,t}^{\text{policy}}\). Figure 3: Design diagram of the UAV swarm intelligence control experimental system. The manager influences the UAV swarm's decision-making through the control strategy and adjusts the reward mechanism to optimize overall behavior. The manager acts at a higher decision-making level and influences the swarm's behavior by adjusting the reward structure and providing strategic policy guidance. The manager observes the global map state, task location, and UAV rewards. Its output is the probability distribution of its control strategy, as shown in the following formula:

\[\tau\sim\pi_{i}\Big{(}o_{c,t}^{\text{vision}}\,,o_{c,t}^{\text{outer}}\,,o_{c,t }^{\text{policy}}\,,h_{c,t-1;}\varnothing\Big{)} \tag{10}\]

where the strategy network simultaneously updates the manager's hidden state, \(h_{c,t-1}\), with inputs, including the rewards and task information of all UAVs in the swarm, \(o_{c,t}^{outer}\), and the control strategy from the start of the system to its implementation, \(o_{c,t}^{\text{policy}}\). ### Experimental System Setting

In the UAV swarm control system, the cyclical game between the manager and the UAVs may lead to two trends: (1) intensified competition, reducing task completion efficiency, harming UAV income, and threatening the swarm's sustainable development; (2) insufficient task completion amount, leading to a decline in swarm performance.