# Research on Multi-Robot Formation Control Based on MATD3 Algorithm
## 3 Multi-Robot Formation Control Based on Multi-Agent Reinforcement Learning
### Combining Prioritized Experience Replay
To improve the stability of training, we normalized the weights and derived them as follows:  
\[\omega_{k}=\frac{\left(l*\bar{p}_{k}\right)^{-\beta}}{\max_{m}(\omega_{m})}= \frac{\left(l*\bar{p}_{k}\right)^{-\beta}}{\max_{m}\left(\left(l*\bar{p}_{m} \right)^{-\beta}\right)}=\frac{\left(\bar{p}_{k}\right)^{-\beta}}{\max_{m} \left(\left(\bar{p}_{m}\right)^{-\beta}\right)} \tag{20}\]  
where \(m\) is the sample ordinal number with the largest weight. Therefore, the two critic network loss functions in the MATD3 algorithm can be rewritten as:  
\[J\left(\theta_{i}^{Q_{1}}\right)=\mathbb{E}_{q^{t},q^{t},q^{t},q^{t+1}}\left[ \omega_{k}*\left(Q_{1}\left(o_{1}^{t},\ldots,o_{N}^{t},a_{1}^{t},\ldots,a_{N} ^{t};\theta_{i}^{Q_{1}}\right)-y_{i}^{t}\right)^{2}\right] \tag{21}\]  
\[J\left(\theta_{i}^{Q_{2}}\right)=\mathbb{E}_{o^{t},q^{t},q^{t},q^{t+1}}\left[ \omega_{k}*\left(Q_{2}\left(o_{1}^{t},\ldots,o_{N}^{t},a_{1}^{t},\ldots,a_{N} ^{t};\theta_{i}^{Q_{2}}\right)-y_{i}^{t}\right)^{2}\right] \tag{22}\]  
where \(\omega_{k}\) is the sampling weight of sample \(k\). \(y_{i}^{t}\) is the target \(Q\) value of robot \(\mathbf{r}_{i}\), and \(Q_{j}\left(o_{1}^{t},\ldots,o_{N}^{t},a_{1}^{t},\ldots,a_{N}^{t};\theta_{i}^{ Q_{j}}\right);j=1,2\) is the estimated \(Q\) value of the critic network. The flow of the MATD3 algorithm based on prioritized experience replay is shown in Algorithm 1. Steps 1 and 2 initialize the network parameters of each robot and the corresponding target network parameters. Step 3 initializes the training parameters, including the conditioning coefficients \(\alpha\) and sampling weight coefficients \(\beta\), the experience buffer \(D\), the number of experiences \(K\) for each extraction, and the network update frequency \(d\). Steps 4 to 26 introduce the framework process of the MATD3 algorithm for the PER, as shown in Section 3.1. \(o\) denotes the set of all robot observations. \(a_{i}\) denotes the action of robot \(\mathbf{r}_{i}\), where \(\mathcal{N}(0,\sigma)\) is the random noise in the training. MATD3 calculates the priority \(p_{k}\) and sampling weight \(\omega_{k}\) for each experience during sampling according to the formula in Section 3.3 and subsequently updates the critic network and actor network. Finally, a soft update of the parameters of the target network is performed. ```
1: Create actor network \(\mu\) and two critic networks \(Q_{1}\), \(Q_{2}\) by using \(\theta_{i}^{\mu}\), \(\theta_{i}^{Q_{1}}\), and \(\theta_{i}^{Q_{2}}\)
2: Initialize target \(\mu\), target \(Q_{1}\), and target \(Q_{2}\) by using \(\hat{\theta}_{i}^{\mu}\), \(\hat{\theta}_{i}^{Q_{1}}\), and \(\hat{\theta}_{i}^{Q_{2}}\)
3: Initialize priority parameters \(\alpha\), \(\beta\); replay buffer \(D\); minibatch \(K\); update policy frequency \(d\)
4:for episode=1 to total episode do
5: Generate random noise for action exploration
6: Acquire original observations \(o=(o_{1},...,o_{N})\)
7:for step=1 to max episode length do
8: for each robot \(i\), select actions \(a_{i}=\mu_{i}(o_{i})+\text{clip}(\mathcal{N}(0,\sigma),-c_{i},c_{i})\)
9: Execute actions \((a_{1}^{t},...,a_{N}^{t})\); observe \(r_{i}^{t}\) and \(o_{i}^{t+1}\)
10: Store \((o^{t},a_{1}^{t},...,a_{N}^{t},r^{t},o^{t+1})\) in \(D\)
11:\(s^{t}=s^{t+1}\)
12:for robot \(i=1\) to \(N\)do
13:for\(j=1\) to \(K\)do
14: Sample experience \(k\) with probability \(p_{k}\) from \(D\)
15: Compute weight \(\omega_{k}\) and TD-error \(\delta_{k}\)
16: Update the priority of \(k\) according to \(|\delta_{k}|\)
17:endfor
18: Calculate target critic Q value via (7)
19: Update critic by minimizing the loss via (21), (22)
20:if\(t\%d=0\)then
21: Update policy \(\mu_{i}\) with gradient via (13)
22:endif
23:endfor
24: Use (4) and (6) soft update target network parameters \(\hat{\theta}_{i}^{\mu}\), \(\hat{\theta}_{i}^{Q_{1}}\), and \(\hat{\theta}_{i}^{Q_{2}}\)
25:endfor
```  
**Algorithm 1** MATD3 with prioritized experience replay.