# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Bi-Directional Dynamic Interaction Process and Evolutionary Game Mechanism  
The double-layer reinforcement learning framework provides a structured learning approach to simultaneously optimize the behavior of inner agents and the control strategy of the outer manager, thereby enhancing the system's comprehensive task effectiveness. This framework integrates inner and outer loop learning, with the strategy network acquiring knowledge through experience, without relying on prior modeling knowledge or assumptions. The specific details are provided in Algorithm 1. ```
1:Initialization \(\{\}\): \(\{\}\)
2:Initialization \(\{\}\): \(\{\}\)
3:Initialization \(\{\}\): \(\{\}\)
4:Initialization \(\{\}\): \(\{\


```
Input: Initial solution, sampling range \(H\), number of control policy change periods \(T\), stopping criteria for policy optimization \(C\), state, observation values, and hidden vectors \(s,o,h,o_{c},h_{c}\) Output: Trained agent policy network weights \(\theta\), trained manager policy network \(\varnothing\)
1. Initialization: Agent and manager policy weights \(\theta,\varnothing\Leftarrow\theta_{0},\varnothing_{0}\), agent and manager data buffers \(D,D_{C}\Leftarrow\{\},\{\}\)
2. **while** network policy weights are being trained **do**
3. **for**\(t=1\) to \(H\)do**
4. \(\mathbf{a},\mathbf{h}\Leftarrow\pi_{i}(.\mid o,h,\theta)\) agent makes a decision, executes the action, and updates the agent's hidden layer parameters
5. **if**\(t\bmod T=0\)then**
6. \(\tau,h_{c}\Leftarrow\pi_{c}(.\mid o_{c},h_{c},\theta)\) adjust the control policy and update the manager's hidden layer parameters
7. **else**
8. \(h_{c}\Leftarrow\pi_{c}(.\mid o_{c},h_{c},\theta)\) control policy remains unchanged, update the manager's hidden layer parameters
9. **endif**
10. \(D\Leftarrow D\cup\{(o,a,r,o^{\prime})\}\)_update_ agent's data buffer
11. \(D_{C}\Leftarrow D_{C}\cup\left\{\left(o_{c},\tau,r_{c},o^{{}^{\prime}}_{c} \right)\right\}\) update manager's data buffer
12 \(\{s,o,o_{c}\}\Leftarrow\left\{s^{{}^{\prime}},o^{{}^{\prime}}_{c}\right\}\) update manager's state
13 endfor
14. \(D,D_{C}\Leftarrow\{\},\{\}\) update \(\theta,\varnothing\) using buffer data and reset the buffer data
15 if the training cycle ends then
16. \(s,o,h,o_{c},h_{c}\Leftarrow s_{0},o_{0},h_{0},o_{C,0},h_{c,0}\) reset training cycle
17. **endif**
18. **if** stopping criteria are met, **then**
19. **Return trained agent and planner policy network parameters \(\theta\) and \(\varnothing\)
20. **endif**
21. **endwhile**
```

**Algorithm 1**Design of Swarm Intelligence Control Based on Double-layer Deep Reinforcement Learning

In the inner loop, the agent optimizes its behavior by completing tasks and receiving rewards, thus maximizing the objective function. Due to the dynamic changes in the manager's control strategy, the agent operates in a non-stationary Markov decision environment, requiring continuous adaptation to environmental changes and behavior adjustment to respond to new strategies. In the outer loop, the conflict between the agents in the swarm, each seeking to maximize its benefits, and the system's overall objectives forces the manager to periodically adjust the control strategy according to the learning objectives, thereby improving comprehensive task effectiveness. Changes in agent behavior induce non-stationarity issues for the manager, requiring joint training and synchronous updates of both the agents' and the manager's policy weights. In the double-layer reinforcement learning framework, the collaborative optimization between the agents and manager, as well as extreme control strategies, may lead to non-stationarity and partial observability issues during the learning process, causing both agents and managers to deviate from optimal behavior. To solve these issues, this paper usesphase learning, entropy regularization, and simulated annealing algorithms to stabilize the learning process [37, 38].