## 3 Bio-Inspired Fission-Fusion Control and Planning via Reinforcement Learning Algorithm
### Sub-Swarm Confrontation via Reinforcement Learning
#### 4.3.1 Evaluation of the Probabilistic Starling-Inspired Topological Structure

For the validation of the PSTI structure, we conducted multiple tests with varying birth ranges for the agents (ranging from 4 to 13). Each test consisted of 1000 trials, with 130 iterations per trial. Figure 3 displays the probabilities of generating a stable swarm using the previous seven-nearest-neighbor and our proposed PSTI structures, respectively, after randomly generating 20 agents at different birth ranges. From Figure 3, it is observed that both methods achieve high swarm rates within a small range. However, as the range increases, the number of seven-nearest-neighbour topological interaction models gradually exhibits local convergence problems. Even at a range of 15, the probability of local convergence reduces to 50 percent. Conversely, the proposed PSTI structure consistently produces superior results across all tested ranges, which demonstrates its ability to enhance the swarming efficiency of UAV swarms while reducing the occurrence of local convergence. It is worth noting that the PSTI structure does exhibit some instances of local convergence due to the random characteristics of the incorporated probability coefficients. However, we can maintain a low communication load while achieving superior swarming performance with our proposed approach by balancing the probability factors. #### 4.3.2 Simulation of the Sub-Swarm Confrontation via Reinforcement Learning Algorithm

We build a simulation environment to satisfy the demand for abundant interactions between the swarm and the dynamic obstacle. Assuming that the mission area is \(100\times 100\). The coordinates of the parent swarm, sub-swarm, and dynamic obstacle are generated by the planning algorithms mentioned above. We use the center point of the sub-swarm to optimize its trajectory, and the sub-swarm flies at a constant altitude of its initial height. When an episode starts, the coordinates of both swarms and the dynamic obstacle are generated. Once the parent swarm is captured by the dynamic obstacle, when \(d_{c2p}\leq d_{c2p}^{min}\) or the sub-swarm runs for more than \(\mathcal{T}=28\) steps, the episode ends. The maximal speed of the parent swarm, sub-swarm, and the dynamic obstacle are \(v_{p}^{max}=1\), \(v_{sub}^{max}=2\), \(v_{e}^{max}=1.5\), respectively. We then evaluate the performance of RL-based algorithms in the simulation environment, including the PPO, SAC, and DQN methods. For the sake of inadaptability to the continuous action space of the DQN algorithm, we discretize each action of sub-swarm (i.e., the flying speed (\(v_{t}\)) and direction (\(\theta_{t}\))) into 10. The three algorithms have the same

Figure 3: Swarm effect under different topological interactions. hyperparameters of \(\gamma=0.99\), a learning rate of \(l_{r}=0.0003\), and network architecture of \(\mathcal{N}=128\times 128\times 128\). Figure 4 shows the accumulative reward of these RL algorithms over training steps for the same network depth. It can be observed that these algorithms achieve similar accumulated rewards, while the agent based on the SAC algorithm converges faster and is as stable as the agent of the PPO algorithm. Figure 5 illustrates the trajectory of the sub-swarm planned by using the PPO algorithm. The visualization demonstrates the efficient response of the sub-swarm to dynamic obstacles, considering various obstacle coordinates. The sub-swarm consistently approaches the target successfully by the end of each episode. Figure 4: Accumulated reward versus training steps of different RL algorithms. Figure 5: Planned path of sub-swarm in the situation of the different initial coordinates of the dynamic obstacle, including (**a**) (22.62, 21.39), (**b**) (23.31, 27.70), (**c**) (26.21, 26.35), and (**d**) (29.50, 21.71), respectively. 3.3 Simulation of the Bio-Inspired Fission-Fusion Control and Planning via Reinforcement Learning Algorithm

Simulation results of swarm motion are depicted in Figure 6a-f.