# Strategies for Scaleable Communication and Coordination in Multi-Agent (UAV) Systems
###### Abstract  
A system is considered in which agents (UAVs) must cooperatively discover interest-points (i.e., burning trees, geographical features) evolving over a grid. The objective is to locate as many interest-points as possible in the shortest possible time frame. There are two main problems: a control problem, where agents must collectively determine the optimal action, and a communication problem, where agents must share their local states and infer a common global state. Both problems become intractable when the number of agents is large. This survey/concept paper curates a broad selection of work in the literature pointing to a possible solution; a unified control/communication architecture within the framework of reinforcement learning. Two components of this architecture are locally interactive structure in the state-space, and hierarchical multi-level clustering for system-wide communication. The former mitigates the complexity of the control problem and the latter adapts to fundamental throughput constraints in wireless networks. The challenges of applying reinforcement learning to multi-agent systems are discussed. The role of clustering is explored in multi-agent communication. Research directions are suggested to unify these components. reinforcement learning; multi-agent systems; distributed systems; clustering; mobile ad-hoc networks; tracking; belief propagation; computational complexity; scaling laws +
Footnote †: journal:  
0000-0001-1999-22000  
communicate by routing messages in multi-hop fashion from one agent to the next. Their relative positions change as they move, which alters the underlying network topology and the implied communication routes. They must constantly update their views of the network topology and refresh their forwarding tables, all while sharing the local states required for coordination. Only after the local states have been shared, are agents able to infer the global state and coordinate their actions. However, even with universal knowledge of the global state, the optimal control policy is still ambiguous. Agents could behave selfishly (or \"greedily\") to maximize their \"take\" of interest-points, or cooperatively, favoring the benefit to the collective over the individual. They could act strategically, to obtain future returns at the expense of the present. The dynamics of the interest-points, initially unknown, also factor into these decisions. Ultimately, the decision space grows exponentially in size (or worse) with the number of agents, which makes the optimal solution unscaleable. The control and communication problems are often separate areas of research, but there have been efforts to integrate them in the field of robotics. This survey/concept paper highlights important contributions in both areas through the lens of multi-agent tracking. Some original ideas are also presented to offer perspective on a joint control/communication architecture. The paper is organized as follows. Section 2 reviews the basics of reinforcement learning (RL) with an eye towards multi-agent tracking systems. Section 3 explores the control problem; the complexity of computing control policies in multi-agent RL and the challenge of verifying whether or not these policies are optimal. Methods that exploit locally interactive structure are discussed to alleviate these challenges. Section 4 explores aspects of the communication problem: the fundamental constraints on throughput in wireless networks and the unreliable nature of wireless communication. Multi-level clustering strategies are surveyed as a possible antidote. Section 5 discusses whether the selected works in this survey fit into a unified architecture that encompasses both the control  
Figure 1: A multi-UAV tracking scenario. (**a**) Burning trees and UAVs correspond to interest-points and agents respectively. (**b**) The system must coordinate to visit as many interest-points as possible in the shortest possible time. (**c**) The system state specifies the locations of agents and interest-points.