# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control  
### Overall Architecture of Double-Layer Deep Reinforcement Learning Framework  
To facilitate readers to understand the symbols used in this paper and their meanings, a symbol comparison (Table 2) is presented below for quick reference in the subsequent method description. Swarm intelligence control involves multi-level decision optimization, including individual-level autonomous decision-making, group-level collaborative interaction, and system-level macro-control. In complex dynamic environments, general multi-agent systems are constrained by local information and fail to achieve a global optimal decision. Consequently, a higher-level manager is often introduced to control swarm behavior, optimize resource allocation, and enhance task effectiveness [31]. The method proposed in this paper uses double-layer deep reinforcement learning to construct a hierarchical architecture consisting of inner-layer execution modules (i.e., agents) and outer-layer coordination modules (i.e., manager). Each layer predicts the expected reward based on real-time feedback from the environment and optimizes the behavior strategy accordingly. In complex dynamic environments, multi-agents can be flexibly deployed as needed to accommodate control tasks of different scales and complexities. The overall model architecture is illustrated in Figure 2 and primarily includes the following three core mechanisms:  
(1) Inner-layer agent optimization mechanism: This mechanism focuses on the optimization of autonomous decision-making at the individual level and collaborative optimization at the group level, covering the following aspects:  
(a) Input Information (labeled as 1 and 2):  
* Each agent extracts spatial input values from the environment within its field of view. The input values are generated by the swarm control system map and represent local information, such as task location and environment status. * The agent network input value consists of public observations, manager and agent observations, and agent observations. Public observations provide the swarm control system status and manager control strategy; manager and agent observations include task income, task amount, and swarm income; and agent observations include energy cost, ability, and local space information. This input integrates global and local information to provide a basis for optimization decisions for the policy network. (b) Optimizing Strategy Network (labeled as 3):  
* These spatial inputs are processed through a Convolutional Neural Network (CNN) to extract spatial features. Extracted features are fed into a Deep Neural Network (DNN) and Long Short-Term Memory (LSTM) network for feature \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Symbol** & **Meaning** & **Symbol** & **Meaning** \\ \hline \(t\) & Time & \(i,j,k\) & Agent indices \\ \(\theta\) & Agent policy weights & _Task_ & Task completion amount \\ \(\varnothing\) & Manager policy weights & \(\omega\) & Collision penalty weight \\ \(a_{i}\) & Agent \(i\) action & _TCE_ & Task completion efficiency \\ \(\tau\) & Manager action & _CTE_ & Comprehensive task effectiveness \\ \(s_{i}\) & Agent \(i\) state & _Reward_ & Agent income \\ \(s_{c}\) & Manager state & _Cost_ & Energy cost \\ \(o_{i}\) & Agent \(i\) observation & _Revenue_ & Total funds \\ \(o_{c}\) & Manager observation & \(\beta\) & The ratio of basic income to performance incentive \\ \(h_{i}\) & Agent \(i\) hidden state & _Proportion_ & The fraction of tasks completed relative to the total tasks \\ \(h_{c}\) & Manager hidden state & \(H\) & Sampling horizon \\ \(r_{i}\) & Agent \(i\) reward & \(N\) & The number of agents \\ \(r_{c}\) & Manager reward & \(\eta\) & Degree of nonlinearity \\ \(\pi_{i}\) & Agent \(i\) policy & \(d_{safe}\) & Safety distance \\ \(\pi_{c}\) & Manager policy & \(\gamma\) & Discount factor \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Mathematical symbols and their corresponding meanings. fusion and time-series prediction.