# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Overall Architecture of Double-Layer Deep Reinforcement Learning Framework
This enables agents to optimize their decision-making processes. * The Agent Reinforcement Learning Optimizer adjusts the action strategy based on the output of the policy optimization network. * To enhance task efficiency, multi-agents share local information, enabling collaborative interactions. * Agent Actions (labeled as 4):
* The inner policy network finally outputs specific action instructions, such as search, move, complete tasks, etc. This behavior acts on the environment or task scenario. * Feedback Rewards (labeled as 5):
* When the agent performs an action, the environment provides a certain reward or penalty signal to update the parameters of the inner strategy and complete the closed loop of reinforcement learning. The inner-layer agent optimization mechanism ensures effective decision-making at the individual level while achieving a group-level collaboration. Detailed explanations are provided in Section 3.2. (2) Outer-layer manager optimization mechanism: This mechanism focuses on macro-control at the system level, responsible for optimizing global resource allocation and control strategies:  
Figure 2: Architecture diagram of the swarm intelligence control design method based on double-layer deep reinforcement learning. 1. Input Information (labeled as 1 and 2): * The manager extracts global input values from the observation data of the environment and the agent, including the state of the environment, the distribution of tasks, and the action state of the agent. This information is integrated into the manager network input value as the input for subsequent feature extraction. * The manager network input values include public, manager, and agent observations. Compared with the agent network input value, there is no agent observation. This input integrates manager observations, observed environmental states, global information, and agent observations, providing a basis for the optimization decision of the policy network. 2. Optimizing Strategy Network (labeled as 3): * The manager comprehensively analyzes the environmental state, global information, and observable values of the agents. The CNN is used to extract features from the environment and agent distribution, followed by feature fusion and time series prediction using the DNN and LSTM to optimize resource allocation and control strategies. * The manager reinforcement learning optimizer captures the dynamic changes in the swarm in both temporal and spatial dimensions. Based on this, it predicts swarm behavior trends, thereby enabling the efficient optimization of control strategies and resource allocation. 3. Manager Actions (labeled as 4): * The manager generates multiple control strategies (e.g., strategy 1, strategy 2, \(\ldots\), strategy N) and selects the optimal one to guide agent behavior. 4. Feedback Rewards (labeled as 5): * The manager also receives feedback rewards based on global performance indicators and multi-agent collaboration effects, thereby continuously optimizing its decision-making strategy. The outer-layer manager optimization mechanism ensures comprehensive system-level control by utilizing global information to optimize task allocation and strategy formulation, thereby enhancing the comprehensive task effectiveness of the system. Detailed explanations are provided in Section 3.3. (3) Bilateral dynamic interaction and evolutionary game mechanism: This mechanism lies at the intersection of the inner-layer agents and the outer-layer manager, enabling dynamic collaboration and optimization: 1. Bilateral Dynamic Interaction: * The manager formulates control strategies based on the global state to guide agents' decision-making. After executing tasks, agents provide feedback on local states, influencing the manager's strategy adjustments. Additionally, agents share information through local interactions to enhance collaborative effects. 2. Evolutionary Game Mechanism: * Agents adjust their actions in dynamic environments based on reward maximization to adapt to optimal task allocation and resource utilization.