# Strategies for Scaleable Communication and Coordination in Multi-Agent (UAV) Systems
## 5 A Proposed Control/Communication Architecture for Multi-Agent Systems
### Developing a Proof-Of-Concept: A Structured Approach  
The architecture in Figure 8 depicts a plausible strategy for the multi-agent tracking problem, based on approaches to some of the sub-problems in the literature. A proof-of-concept/implementation for the whole architecture has yet to appear, but the goal of this paper is to motivate efforts to that end. The following section outlines a structured approach towards a possible implementation, using insights from the authors' experiences in this space. The main objective of this architecture is to support a scaleable solution; one that enables \"approximately\" optimal tracking with arbitrarily many agents. If we assume all agents have access to full state information, the complexity of multi-agent reinforcement learning still grows exponentially in the number of agents (without any mitigating strategies). It is natural to address this bottleneck first and progressively relax each assumption, adding more modules in the architecture until reaching a fully decentralized solution. Some reasonable guidelines define an approach towards a proof-of-concept. The first is to keep the action-space small; the complexity of a primitive space (i.e., \"up\", \"down\", \"left\", \"right\"), still grows exponentially in the number of agents and remains unsolved. Once a solution for the primitive space is found, more complex aerodynamic actions can be considered. The next is to start with basic RL before proceeding to deep RL. The former works with the \"raw\" Q-tables \(Q_{\pi}(s,\mathbf{a})\) and does not include any function approximation. Since the number of entries in the table (state-action pairs) grows exponentially in the number of agents and interest-points, basic RL is limited to small numbers of agents and interest-points and grid sizes. Nevertheless, it is useful to understand the reward function in small dimensions, for approximation in large dimensions with neural networks. The overarching principle is to extrapolate more complex behavior of the system from its simpler forms, proceeding in stages, gently increasing the complexity of the system while adding forensic tools in software to help identify if and where divergence occurs. We propose a set of milestones consistent with the guidelines above, that define a structured approach towards a proof-of-concept. While there is no one fixed set of milestones, the approach we propose may inspire others. 1. _The Single-Agent, Stationary Interest-Points Scenario._ The agent visits a sequence of stationary interest-points on a two-dimensional grid, using the shortest path. Training is performed with basic RL, under the assumption the agent has full-state state information. For a single stationary interest-point, this scenario reduces to the grid-world problem, a well-known example in RL. The grid-world problem becomes much harder with two interest-points. A scenario with multiple interest-points corresponds to the TSP, where the optimal solution requires an exhaustive search over all possible trajectories. The next section will explain why this milestone, even with two interest-points, benefits from advanced strategies like hierarchical RL. 2. _The Multi-Agent, Stationary Interest-Points Scenario._ A system of multiple agents finds an approximately optimal set of trajectories that visit stationary interest-points on a two-dimensional grid. Training is performed using basic RL under the assumption that each agent now observes the full system state. This milestone incorporates macro-actions (i.e., \"Agents 1, 4, and 5 move to sub-region A\") to keep the distribution of agents and interest-points in equilibrium. It focuses on policies reasonably achievable by training Q-tables, excluding moving interest-points and the large scale activity of many agents (i.e., dynamic sky-way networks). 3. _The Multi-Agent Mobile Interest-Points Scenario._ A system of agents tracks moving interest-points on a two-dimensional grid. Training is performed using deep RL, under the assumption that each agent observes the full system state.