# Research on Multi-Robot Formation Control Based on MATD3 Algorithm
## 1 Introduction
On the other hand, most of the reinforcement learning algorithms used in the existing literature are DDPG algorithms [25], which originated from the DQN algorithm [26], which is an off-policy approach. The DDPG algorithm updates the target value function during training using the action that is currently considered to have the largest Q value, which can lead to the problem of Q overestimation. However, the overestimation is usually non-uniform, leading to unreliable decisions made by the algorithm. Based on these problems, this paper investigated multi-robot formation navigation in environments with obstacles using deep reinforcement learning in conjunction with the leader-following method. Based on the twin delayed deep deterministic policy gradient (TD3) algorithm [27] combined with the CTDE framework [28], an improved multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm is proposed. The main contributions of this paper are as follows: (1) To address the overestimation problem in the DDPG algorithm, we used the TD3 algorithm in combination with the CTDE framework to implement multi-robot formation control. (2) We designed a hybrid reward mechanism that allows the robot formation to change its formation reasonably when facing an obstacle area and thus pass. (3) The prioritized experience replay (PER) mechanism [29] was also added to make full use of high-quality samples for training and improve the learning efficiency. At the same time, we set up both guided rewards and sparse rewards to facilitate robot learning. The simulation results showed that the algorithm can perform the formation navigation task in environments with obstacles well. The rest of the paper is organized as follows: In Section 2, we provide the problem description and system modeling. Section 3 describes the reinforcement learning algorithm used in this paper and the improvements made. Section 4 describes the reinforcement-learning-based formation control scheme, including the state space design and reward function design. In Section 5, we conduct the simulation experiments and analyze the experimental results. Finally, Section 6 concludes the whole paper. In this paper, we assumed that the leader robot can globally perceive the location of the target point. Define the detection distance of the robot as \(\rho_{d}>0\). When the distance between the robot and the obstacle is less than \(\rho_{d}\), the robot can sense the obstacle's position. During the formation movement, the robot can obtain the status information of itself and other robots in real-time, which includes velocity information \(v_{i}^{t}\) and position information \(p_{i}^{t}\). The control target of formation navigation is divided into three parts: maintaining the default formation, avoiding collision, and reaching the target position. (1) Maintaining the default formation: The formation members start from their respective starting positions and move to form a default formation, and the default spacing should be maintained between all robots. Therefore, the control objective is to minimize the distance error between robots:  
\[\min\left|d_{i}^{j}-d_{i}^{j^{*}}\right|,i,j\in[1,\ldots,N],i\
eq j \tag{2}\]  
where \(d_{i}^{j^{*}}\) is the desired relative distance between robot \(\mathbf{r}_{i}\) and robot \(\mathbf{r}_{j}\) and \(d_{i}^{j}\) is the actual relative distance between robot \(\mathbf{r}_{i}\) and robot \(\mathbf{r}_{j}\). (2) Collision avoidance: During the formation movement, the distance between robots and between robots and obstacles should satisfy \(d_{i}^{j}\geq\rho_{r}\) and \(dis\left(\mathbf{r}_{i},\mathbf{o}_{j}\right)\geq\rho_{o}\), where \(dis(\cdot)\) denotes the Euclidean distance in two dimensions and \(\rho_{r}\) and \(\rho_{o}\left(\rho_{d}>\rho_{o}\right)\) are the safe distance between robots and the safe distance between robots and obstacles, respectively.