## 3 Bio-Inspired Fission-Fusion Control and Planning via Reinforcement Learning Algorithm
### Sub-Swarm Confrontation via Reinforcement Learning  
After splitting the sub-swarms, we formulate the sub-swarm path planning problem with a Markov decision process (MDP). An MDP provides a mathematical framework for modeling sequential decision-making problems where an agent must choose actions in a sequence to achieve a desired goal. The MDP can be denoted as a 4-tuple: \(\langle S,\ A,\ P,\ R\rangle\), where \(S\) is the set of state \(s_{t}\), \(A\) is the set of available actions (\(a_{t}\)) of the UAV, \(P\) is the transition probability distribution, and \(R\) defined as \(S\times A\to R\), which is the reward function. In time slot \(t\), the state \(S\) in the environment can be denoted as \(S_{t}=\big{\{}c_{ps},\ c_{ss},\ c_{e},\ c_{t}\big{\}}\), which indicates the coordinate of the parent swarm, sub-swarm, dynamic obstacle, and target, respectively. The action of a UAV can be written as \(A_{t}=\{v_{t},\ \theta_{t}\}\), which denote the flying speed and direction of the UAV, respectively. The reward function in time slot \(t\) is comprised of three parts, which are described as  
\[R_{t}=r_{s2e}+r_{s2t}+r_{e2p} \tag{15}\]  
The first term \(r_{s2e}\) is a reward for the distance between the sub-swarm and the dynamic obstacle, denoted as  
\[r_{s2e}=2\times\left(e^{-\frac{(d_{s2e}-r_{s2e}^{\alpha})^{2}}{2\sigma^{2}}}- 0.5\right)+\left\{ \begin{array}{rl}\alpha\log(\frac{d_{s2e}}{d_{s2e}^{\alpha }}),&d_{s2e}<d_{s2e}^{safe}\\ 0,&d_{s2e}\geq d_{s2e}^{safe}\end{array} \right. \tag{16}\]where \(d_{s2e}\) is the distance between the sub-swarm and the dynamic obstacle, \(d_{s2e}^{safe}\) and \(d_{s2e}^{capture}\) are the safe and capture distance between the sub-swarm and dynamic obstacle, respectively. This term is parameterized through \(\sigma\) and \(\alpha\). The second term (\(r_{s2t}\)) is a reward for the distance between the sub-swarm and the target, given by  
\[r_{s2t}=-\beta d_{s2t}+c_{1} \tag{17}\]  
where \(d_{s2t}\) is the distance between the sub-swarm and the target, and \(\beta\) and \(c_{1}\) are parameters for adjustment. The third term (\(r_{s2p}\)) is a reward for the distance between the dynamic obstacle and the parent swarm, and it is described as  
\[r_{e2p}=\left\{ \begin{array}{rl}\gamma clip(\log\left(\frac{d_{d2p}}{d_{s2p} ^{min}}-1\right),-1,\ 1),&d_{e2p}>d_{e2p}^{min}\\ c_{2},&d_{e2p}\leq d_{e2p}^{min}\end{array} \right. \tag{18}\]  
where \(d_{e2p}\) means the distance between the dynamic obstacle and the parent swarm, \(d_{e2p}^{min}\) means the minimum distance between the dynamic obstacle and parent swarm, \(\gamma\) and \(c_{2}\) are parameters for adjustment, and \(clip\left(\log\left(\frac{d_{d2p}}{d_{s2p}^{min}}-1\right),-1,\ 1\right)\) is a mathematical operation which removes the incentive for moving \(\log\left(\frac{d_{d2p}}{d_{s2p}^{min}}-1\right)\) outside of the interval \([-1,\ 1]\). The sub-swarm is captured by the dynamic obstacle if \(d_{e2p}<=d_{e2p}^{min}\). To address the path planning problem of the sub-swarm, we propose an RL-based algorithm. Reinforcement learning is an effective approach for tackling sequential decision problems in MDPs, aiming to maximize the cumulative reward within an episode. Among various RL algorithms, the proximal policy optimization (PPO) algorithm stands out as one of the most efficient methods for policy optimization. Another notable off-policy RL algorithm is the soft actor-critic (SAC) algorithm, which demonstrates high sample efficiency. Additionally, deep Q-learning (DQN) is a widely used RL method, which, however, is limited to discrete action spaces. In this study, we introduce the SAC algorithm to realize the antimotion of sub-swarm with dynamic obstacles.