## 2 Background
### Multi-Agent Systems  
Multi-agent systems (MASs) consist of multiple interacting agents that work together to achieve a common goal. These systems are characterized by their ability to operate decentralized, where each agent makes decisions based on local information and interactions  
Figure 4: Block diagram showing the position control problem with obstacle avoidance. with other agents. This approach offers several advantages, including increased robustness, scalability, and flexibility [32]. In cooperative multi-agent systems, agents interact together to maximize a shared objective. This requires effective communication and coordination among agents to ensure that their actions are aligned toward the common goal. Examples of cooperative MASs include swarm robotics [33], where multiple robots collaborate to perform tasks such as search and rescue [34], environmental monitoring, and formation control. To implement cooperation among agents, the interaction can be established in two different ways: (1) all agents have the same hierarchy/rank, and (2) there is a leader or master and the rest are followers. In the first approach, known as egalitarian cooperation, all agents collaborate equally, sharing information and decisions to achieve a common goal. This model promotes robustness and flexibility, as it does not rely on a single point of failure. In the second approach, known as hierarchical cooperation, a leader or master agent coordinates the actions of the other agents, who act as followers. This model can be more efficient in terms of decision-making and coordination, especially in complex tasks that require centralized direction. Both approaches have their advantages and challenges, and the choice between them depends on the nature of the task and the environment in which the agents operate. To make a formation with the leader-follower approach, the following equations can be taken into account [35], where \(N\) is the number of followers:  
\[v_{m}(t)=K_{p}E_{pm}(t)-K_{f}E_{f}(t) \tag{6}\]  
\[d_{s}(t)=\sum_{i=1}^{N}d^{f_{i}}(t) \tag{7}\]  
In the case of the followers, they use the position of the leader as a target point and a distance regarding their position in the formation concerning the leader. These equations help in designing the control strategy for the followers to maintain a specific formation relative to the leader. The velocity of the leader (\(v_{m}(t)\)) depends on two terms (Equation (6)): \(K_{p}E_{pm}(t)\) which is its position error (\(E_{pm}(t)\)) and a control gain (\(K_{p}\)) that defines the importance of this error in the control. The second term (\(-K_{f}E_{f}(t)\)) represents the sum of the position error of each follower (\(E_{f}\)) in the formation (Equation (7)), multiplied by a control gain (\(K_{f}\)). By adjusting the control gains (\(K_{p}\), \(K_{f}\)) and desired positions, the formation can be maintained dynamically as the leader moves. Figure 5 shows a representation of non-cooperative multi-agent systems [36] from points \(A\) to \(C\). The leader robot is represented in red and the followers in black. The dotted lines represent the distances between the leader and the follower robots. The experiment starts at point \(A\), where the position of the robots is not in formation, which is reached at point \(C\). All the robots move from the initial position to point \(B\) (dashed lines), as can be seen, the leader does not wait for the followers and the formation is not maintained. Then, the leader arrives at point \(C\) and the followers take time to reach the final formation at point \(C\). On the other hand, the example of cooperative multi-agent systems is shown from point \(C\) to \(E\). The robots maintain the formation during the maneuver. Deep Reinforcement Learning and Multi-Agent Systems  
Multi-agent systems have a wide range of applications across various domains. In robotics, a MAS is used for tasks such as formation control, where multiple robots maintain a specific formation while navigating through an environment.