# Research on Multi-Robot Formation Control Based on MATD3 Algorithm
## 3 Multi-Robot Formation Control Based on Multi-Agent Reinforcement Learning
### Matd3  
Overestimation bias is a common problem in reinforcement learning, where the action value function is maximized and used to learn the optimal policy, which can cause the learned policy to become inaccurate [31]. The TD3 algorithm updates the target value function by learning two critic networks and choosing a smaller estimated \(Q\) value to approach the true \(Q\) value to alleviate the overestimation problem. Furthermore, TD3 uses a policy delayed update approach, where the critic network is updated for a certain number of steps before the actor network is updated [27]. The MATD3 algorithm combines the implementation process of TD3 and the \"centralized training, distributed execution\" architecture. The algorithm structure of MATD3 is designed as shown in Figure 4. MATD3 uses two sets of actor-critic structures, and each robot has its own independent actor network \(\mu\Big{(}o_{i};\theta_{i}^{\mu}\Big{)}\) with parameter \(\theta_{i}^{\mu}\), which calculates action \(a_{i}\) based on the current observation \(o_{i}\) of robot \(\mathbf{r}_{i}\). The two critic networks are trained in a centralized manner, respectively, network \(Q_{1}\Big{(}o_{1},\ldots,o_{N},a_{1},\ldots,a_{N};\theta_{i}^{Q_{1}}\Big{)}\) with parameter \(\theta_{i}^{Q_{1}}\) and network \(Q_{2}\Big{(}o_{1},\ldots,o_{N},a_{1},\ldots,a_{N};\theta_{i}^{Q_{2}}\Big{)}\) with parameter \(\theta_{i}^{Q_{2}}\), which take the observations and actions of all robots as the input, and the network output is the estimated \(Q\) value to evaluate the goodness of action \(a_{i}\), thus improving the performance of the actor network. For stable learning, the target actor network and two target critic networks are constructed with parameters \(\theta_{i}^{\mu}\), \(\theta_{i}^{Q_{1}}\), and \(\hat{\theta}_{i}^{Q_{2}}\), respectively, and the parameter update law of the target network is:  
\[\hat{\theta}_{i}^{\mu}=\tau\theta_{i}^{\mu}+(1-\tau)\hat{\theta}_{i}^{\mu} \tag{4}\]  
\[\hat{\theta}_{i}^{Q_{1}}=\tau\theta_{i}^{Q_{1}}+(1-\tau)\hat{\theta}_{i}^{Q_{ 1}} \tag{5}\]  
\[\hat{\theta}_{i}^{Q_{2}}=\tau\theta_{i}^{Q_{2}}+(1-\tau)\hat{\theta}_{i}^{Q_{ 2}} \tag{6}\]  
where \(\tau\) is the soft update parameter. To alleviate the overestimation problem, a smaller \(Q\) value is chosen to calculate the target \(Q\) value, so that the estimated \(Q\) value can be close to a more realistic and reliable target \(Q\) value, which in turn guides the parameter update of the actor network. The target \(Q\) value of robot \(\mathbf{r}_{i}\) can be defined as follows:  
\[y_{i}^{t}=r_{i}^{t}+\gamma\cdot\min\Bigg{(}Q_{j}\bigg{(}o_{1}^{t+1},\ldots,o_ {N}^{t+1},\hat{a}_{1}^{t+1},\ldots,\hat{a}_{N}^{t+1};\hat{\theta}_{i}^{Q_{j}} \bigg{)}\bigg{)};j=1,2 \tag{7}\]where \(\gamma\) is the discount factor and \(\bar{a}_{i}^{t+1}\) is the target action of the clipping, which is the output of the target actor network with the following expression. \[\bar{a}_{i}^{t+1}=\mu\Big{(}o_{i}^{t+1};\hat{\theta}_{i}^{\mu}\Big{)}+\text{clip }\big{(}\mathcal{N}(0,\sigma),-c_{i},c_{i}\big{)} \tag{8}\]  
where \(\mathcal{N}(0,\sigma)\) denotes a random noise satisfying a normal distribution and \(c_{i}\) is the noise edge value. Define the two critic loss functions of the robot \(\mathbf{r}_{i}\) as \(J\Big{(}\theta_{i}^{Q_{1}}\Big{)}\) and \(J\Big{(}\theta_{i}^{Q_{2}}\Big{)}\), as follows:  
\[J\Big{(}\theta_{i}^{Q_{1}}\Big{)}=\mathbb{E}_{o^{t},a^{t},r^{t},\rho^{t+1}} \Bigg{[}\bigg{(}Q_{1}\Big{(}o_{1}^{t},\ldots,o_{N}^{t},a_{1}^{t},\ldots,a_{N}^ {t};\theta_{i}^{Q_{1}}\Big{)}-y_{i}^{t}\bigg{)}^{2}\Bigg{]} \tag{9}\]  
\[J\Big{(}\theta_{i}^{Q_{2}}\Big{)}=\mathbb{E}_{o^{t},a^{t},r^{t},\rho^{t+1}} \Bigg{[}\bigg{(}Q_{2}\Big{(}o_{1}^{t},\ldots,o_{N}^{t},a_{1}^{t},\ldots,a_{N}^ {t};\theta_{i}^{Q_{2}}\Big{)}-y_{i}^{t}\bigg{)}^{2}\Bigg{]} \tag{10}\]  
The two critic network parameters \(\theta_{i}^{Q_{1}}\) and \(\theta_{i}^{Q_{2}}\) are updated by gradient descent, specifically in Equations (11) and (12).