# Research on Multi-Robot Formation Control Based on MATD3 Algorithm
## 3 Multi-Robot Formation Control Based on Multi-Agent Reinforcement Learning
### Combining Prioritized Experience Replay  
In order to reuse historical experience data, the experience replay mechanism was first introduced in the DQN algorithm to store historical data in the experience replay buffer, which greatly improves the experience utilization [26]. However, the random sampling approach ignores the importance of the experience data, which may lead to the uneven quality of the sampled experience data and, thus, reduce the learning efficiency. To solve the above problem, the prioritized experience replay (PER) method is proposed, and the PER method prioritizes each experience datum according to its importance and is able to select the more important experiences more times when sampling, thus increasing the learning efficiency [29]. The PER method performs well in single-agent algorithms such as DQN and DDPG. To address the characteristics of the multi-agent CTDE framework, this paper used a centralized experience buffer to store the joint information of all the agents and, then, introduced the prioritized experience replay mechanism to prioritize the sampling experience according to the experience quality. On the other hand, TD-error is expressed as the difference between the estimated \(Q\) value and the target \(Q\) value, which is usually used to update the estimate of the critic network, so it can be used as a correction to the estimate, so the TD-error can reflect how well the algorithm assesses the true value of the experience. Samples with larger TD-error values are given higher priority, and conversely, samples with smaller TD-error values are given lower priority. The TD-error derived by robot \(\mathbf{r}_{i}\) for sample \(k\) through two critic networks is shown in Equations (15) and (16). \[\delta_{i,1}^{k}=y_{i}^{t}-Q_{1}\Big{(}o_{1}^{t},\dots,o_{N}^{t},a_{1}^{t}, \dots,a_{N}^{t};\theta_{i}^{Q_{1}}\Big{)} \tag{15}\]  
\[\delta_{i,2}^{k}=y_{i}^{t}-Q_{2}\Big{(}o_{1}^{t},\dots,o_{N}^{t},a_{1}^{t}, \dots,a_{N}^{t};\theta_{i}^{Q_{2}}\Big{)} \tag{16}\]  
In this paper, the larger value between the absolute values of two TD-errors was chosen as the priority of sample \(k\), as shown in Equation (17). \[p_{k}=\max\biggl{(}\left|\delta_{i,1}^{k}\right|_{\prime}\left|\delta_{i,2}^{ k}\right|\biggr{)} \tag{17}\]  
Figure 4: MATD3 algorithm framework. Then, the sampling probability of sample \(k\) can be defined as:  
\[\bar{p}_{k}=\frac{\left(p_{k}\right)^{\alpha}}{\sum_{l}\left(p_{l}\right)^{\alpha}} \tag{18}\]  
where \(\bar{p}_{k}\) is the sampling probability of sample \(k\). \(\left(p_{k}\right)^{\alpha}\) denotes the priority of sample \(k\) in the replay buffer. \(\alpha\) is a hyperparameter from 0 to 1 that affects the priority, where \(\alpha\) is 0, the sampling criterion is random sampling, and \(l\) denotes the number of samples in the replay buffer. Using the PER method would have the problem of changing the state distribution. Before the PER method was proposed, the replay buffer existed to address the data correlation and make the sample data independently and identically distributed. However, using the PER would more frequently replay experiences with higher TD-error values, which would inevitably introduce bias. In order to avoid bias leading to oscillations or even dispersion during training, the importance sampling method is introduced to appropriately reduce the weights of good samples so that the impact of each sample in the gradient descent is the same during training, thus ensuring the convergence of the results. The sampling weights for sample \(k\) are defined as:  
\[\omega_{k}=\frac{1}{l^{\beta}\cdot p_{k}^{\beta}} \tag{19}\]  
where \(\beta\) is a 0 to 1 hyperparameter that determines the impact of improving the PER on the convergence result. The larger \(\beta\) is, the greater the sampling weight of the lower-priority samples will be.