# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Inner-Layer Agent Optimization Mechanism for Individual and Group Control  
The individual and group control algorithm based on deep reinforcement learning in the inner layer constitutes a hybrid approach. The agents continuously compete and evolve through the interaction with the environment to identify the optimal strategy. Simultaneously, a collaboration among the agents ensures the optimization of the swarm's overall performance. The behavioral strategy objective of agent \(i\) is to maximize its expected cumulative discounted reward, as expressed in the following formula:  
\[\forall i:\max\,\mathrm{E}_{a_{i}\sim\pi_{i}a_{i-i}\sim\pi_{-i}s^{\prime}\sim \tau}\left[\sum_{t=1}^{H}\gamma^{t}*\underbrace{Task_{t,i}}_{r_{i,t}}+Task_{1, i}\right] \tag{1}\]  
where the instantaneous reward, \(r_{i,t}\), is the task completion amount, \(Task_{t,i}\), of agent \(i\) from time \(t-1\) to \(t\). The agent's behavioral strategy depends not only on its observations, \(o_{i,t}\), and hidden states, \(h_{i,t}\), but also on the behavior strategy, \(\pi_{-i}\), of other agents and the environment transition, \(\tau\). Under the discount factor \(\gamma\in(0,1)\), the agent maximizes its discounted expected return by utilizing historical data from parallel environments and accumulating the reward, \(r_{i,t}\), through the strategy network parameters, \(\theta_{i,t}\). The joint strategy \(\pi=(\pi_{1},\ldots,\pi_{N})\) represents the set of strategies for all agents, while \(\pi_{-i}=(\pi_{1},\ldots,\pi_{i-1},\pi_{i+1},\ldots,\pi_{N})\) represents the strategy set excluding agent \(i\). During the training process, the Proximal Policy Optimization (PPO) algorithm [32] is used to map observations to actions through the neural network \(\pi_{i}\) (\(a_{i}\)\(|o_{i}\), \(h_{i}\)?\(\theta\)) and update the strategy weights, \(\theta_{i}\). The specific optimization mechanism is described as follows:  
\[\Delta\theta_{i}\propto\,\mathrm{E}_{a_{1}\sim\pi_{1},\ldots,a_{N}\sim\pi_{n}s ^{\prime}\sim\tau}\left[\sum_{t=1}^{H}\gamma^{t}r_{i,t}\
abla_{\theta_{i}}log \pi_{i}(a_{i,t}|o_{i,t};\theta_{i})\right] \tag{2}\]  
where \(\mathbf{N}\) represents the number of agents in the inner layer, which are of various types and have different fields of view and hidden states. This approach is inspired by concepts similar to MADDP [33], using a centralized training, decentralized execution model, where multi-agents of different types share a parameter set, \(\theta\). Each agent's strategy, \(\pi_{i}\) (\(a_{i}|\)\(o_{i}\), \(h_{i}\)?\(\theta\)), is determined based on its observations, \(o_{i}\), and hidden states, \(h_{i}\), enabling cooperation between agents to improve the swarm's task completion efficiency. This mechanism enables agents to learn new behaviors from agents in different regions, facilitating cross-agent behavior sharing and enhancing learning efficiency. Simultaneously, to prevent collisions between different agents in the inner layer, soft constraints are incorporated into the reward function, as demonstrated by the following mechanism [34]:  
\[R_{total}=R_{original}-\omega\cdot\max(0,\ d_{safe}\ -\ \|p_{A}-p_{B}\|_{2})^{2} \tag{3}\]  
where \(R_{original}\) is the original reward function, \(R_{total}\) is the reward function after the constraint is added, and \(\omega\) is the weight used to adjust the importance of collision penalties. \(d_{safe}\) denotes the safety distance, typically set to 0, while \(p_{A}\) and \(p_{B}\) represent the positions of any two agents. Strategy optimization at the individual level focuses on the agent's perception, memory, and decision-making capabilities, highlighting how agents adjust their behavior based on observations and internal states to adapt to both environmental conditions and task requirements. At the group level, it involves both competitive and cooperative interactions. Agents may either compete for the same task or achieve collaborative optimization through information sharing, with strategy selection determined by their perception and decision-making mechanisms.