# Strategies for Scaleable Communication and Coordination in Multi-Agent (UAV) Systems
## 5 A Proposed Control/Communication Architecture for Multi-Agent Systems  
The goal of this paper was to show that a broad selection of work in the literature fits into a proposed control/communication \"architecture\"; an organized set of interacting, independent modules. This architecture is depicted in Figure 8.  
A \"module\" defines an interface with some concrete functionality, while the \"interface\" specifies the inputs and outputs of the desired task. The implementation or mechanism used to execute the task is _not_ specified by an architecture, the expectation being that every implementation is eventually made obsolete by advances in technology. An architecture, by contrast, describes something more resilient and rooted in fundamental constraints of the system itself, constraints that do not change with technology but describe the limits of what technology can do. In the proposed architecture, these constraints are the complexity and network capacity of multi-agent systems. The modules in Figure 8 are less formal and loosely refer to common tasks appearing repeatedly in the literature. Figure 8 highlights a duality between the control and communication components of the architecture. Both components start with a single-agent module (an RL agent or a cluster) and build a multi-level hierarchy over that module. On the control side, the sub-modules of an agent include the reward function and the mechanism for finding optimal primitive policies (i.e., actor-critic or Q-learning). The outer module includes the single-agent sub-module as well as two sub-modules required in hierarchical multi-agent reinforcement learning. The first ensures the system remains stationary over the option-space, and the second determines the structure of the option-space. The latter, by implication, also includes traffic management capabilities and an exploration/exploitation trade-off. The communication side mirrors this progression, from single-level clusters to hierarchical multi-level clustering trees. Single-level clusters include sub-modules for cluster formation, cluster maintenance, and clock synchronization, whereas multi-level clusters implement the merge, split, and hand-off functionality. Multi-level clustering disseminates local state information to all of the agents in the form of \"beliefs\". The mechanism for belief \"sharing\" or \"propagation\" includes some form of data compression. Beliefs shared through the clustering tree must conform to the locally interactive structure of the state space; agents should only exert higher-order influence on any remote agent receiving their compressed and aggregated local states. The module that interprets locally interactive structure on the control side receives the beliefs shared from the communication side. This module constructs a compressed global state for each agent centered at the agent, and feeds this reconstructed state into the agents' control policy, connecting the control and communication components. Each module in the architecture has received attention in the literature, and there is no single implementation that supersedes all others. Incremental improvements at the modular level will likely continue well into the future, but there are fundamental constraints on capacity and complexity that can never be exceeded. Twin pillars of the proposed architecture (i.e., locally interactive structure and multi-level clustering) cater precisely to  
Figure 8: A proposed control and communication architecture for multi-agent tracking systems. these constraints. Locally interactive structure keeps complexity in check, by settling for approximately optimal performance (13) and compressing the state-space (12). Multi-level clustering satisfies throughput constraints by compressing the beliefs hierarchically (21). These operations are mutually beneficial and motivate a yet-to-be-made proof-of-concept for the full architecture. A drawback of deep RL is that the learning process does not always find the optimal policy \(\pi^{*}(s)\).