# Design of Swarm Intelligence Control Based on Double-Layer Deep Reinforcement Learning
## 3 Double-Layer Deep Reinforcement Learning Design for Swarm Intelligence Control
### Bi-Directional Dynamic Interaction Process and Evolutionary Game Mechanism
\begin{table}
\begin{tabular}{c c c c} \hline
**Category** & **Variable** & **Parameter Settings** & **Origin** \\ \hline \multirow{7}{*}{Core} & Number of UAVs (Region 1: Region 2: Region 3) & 6:6:6 & \\  & UAV’s vision & U\((1,4)\) & \\  & Running Time & 1000 & Industrial UAV \\  & UAV’s performance & \(N(0,1)\) & system \\  & Nonlinearity parameter of efficiency & \(\eta=0.21\) & \\  & Single-period revenue & \(Revenue_{t}=20\) & \\  & UAV’s energy cost for movement & \(k*x\), where \(k=1,x\) is the & \\  & UAV’s exploration energy cost & U\((1,3)\) & [27; 28] \\  & UAV’s completion energy cost & U\((10,15)\) & [27; 28] \\  & UAV’s value type & U\((1,3)\) & [28; 29] \\  & Task respawn probability & 0.1 & [29; 30] \\  & Region 1 task generation rule & \(5\times sin(t)+40\) & [30] \\  & Region 2 task generation rule & \(3\times sin(t)+30\) & [30] \\  & Region 3 task generation rule & \(2\times sin(t)+20\) & [30] \\ \hline \end{tabular}
\end{table} 
Table 4: Important parameter settings in the UAV swarm intelligence control system. \begin{table}
\begin{tabular}{c c c} \hline
**Category** & **Variable** & **Parameter Settings** \\ \hline \multirow{7}{*}{General Settings} & Reinforcement learning training algorithm & PPO \\  & First-stage training steps: second-stage training steps (million) & 150:150 \\  & Training duration (million) & 150 \\  & Second-stage manager strategy annealing duration (million) & 37.5 \\  & Second-stage entropy regularization duration (million) & 70 \\  & Value/Policy networks share weights & False \\  & Discount factor & 0.998 \\  & Generalized Advantage Estimation discount parameter & 0.98 \\  & GAE lambda & 0.98 \\  & Gradient clipping norm threshold & 10 \\  & Value function loss coefficient & 0.05 \\  & Whether the advantage is normalized & True \\  & SGD sequence length & 50 \\  & Number of parallel environment replicas & 60 \\  & Number of fully connected layers & 2 \\  & Sampling horizon (steps per replica) & 200 \\  & Episode length & 1000 \\  & Episodes & 12,000 \\  & Epochs & 1000 \\  & Activation function & Relu, Softmax \\  & CPU & 16 \\ \hline \multirow{7}{*}{Agent Settings} & Agent CNN dimension & 128 \\  & Agent LSTM unit size & 128 \\ \cline{1-1}  & Entropy regularization coefficient & 0.025 \\ \cline{1-1}  & Agent learning rate & 0.0003 \\ \cline{1-1}  & All agents share weights & True \\ \hline \end{tabular}
\end{table} 
Table 5: Important parameter settings of D-DRL. ### Experimental Results and Analysis

This paper selected 10 different random seed groups in the experiments. By averaging the results, it analyzed the convergence of the three reinforcement learning methods (DRL, ABC-DRL, and D-DRL) during the training process. Figure 4a,b depict the convergence curves of the mean reward in the inner and outer layers, respectively. These curves not only reflect the trend of reward changes over time, but also correspond to the improvement in task completion amount and comprehensive task effectiveness. It can be observed that all three methods exhibit a gradual increase in rewards followed by stabilization, with D-DRL demonstrating the highest stability and convergence level, particularly in terms of comprehensive task effectiveness. Figure 4c,f illustrate the changes in policy entropy in the inner and outer layers, representing the process of the agent exploring high-value actions. The policy entropy of the D-DRL method is slightly lower and fluctuates less in the later stages, indicating that it locks onto high-value strategies more quickly, thereby reducing policy uncertainty. Figure 4d,e compare the convergence of policy gradient loss, where the gradient loss of all methods shows a downward trend and eventually stabilizes. The D-DRL method exhibits a smoother decline in gradient loss, suggesting more stable policy optimization and the avoidance of large gradient fluctuations. Figure 4g,h analyze the value loss in the inner and outer layers, showing that value loss is initially high due to significant estimation errors in state value assessments.