# Strategies for Scaleable Communication and Coordination in Multi-Agent (UAV) Systems
## 5 A Proposed Control/Communication Architecture for Multi-Agent Systems
### A Case-Study of Milestone M1  
To further explore the process of developing a proof-of-concept, we offer a case-study of milestone M1. First, we introduce a system with one agent and one stationary interest point, then consider some challenges in the extension to two stationary interest-points. The system, depicted in its initial state in Figure 9a, is a grid of side-length 4. The agent objective is to find the shortest path (defined by four primitive actions (\"up\", \"down\", \"left\", \"right\")) from the agent to the interest-point. By inspection, shortest paths have length 4 and are thus simple to verify. The scheme used in this case-study employs basic q-learning (i.e., q-tables). For the single interest-point scenario, the reward \(r\) given to each state-action pair \((s,a)\) is \(r=500\) if the positions of the agent and interest-point coincide, \(r=-10\) if the action drives the agent into the boundary, and \(r=-1\) otherwise. The hyperparameters which determine the update rule are \(\alpha=0.3\), \(\gamma=0.6\), and \(\epsilon=0.1\). The \(\epsilon\)-greedy policy which samples the state space selects a \"greedy\" or \"exploitative\" action \(a^{*}=\max_{a}q(s,a)\) with probability \(1-\epsilon\) and a random exploratory action (uniformly from one of the four primitive choices) with probability \(\epsilon\). After each step, the table of q-values (i.e., the value of each state-action pair) is updated according to the following rule: \(q(s,a)\gets q(s,a)+\alpha(r+\gamma\max_{a}q(s^{\prime},a)-q(s,a))\). Recall that \(s^{\prime}\) is the state outcome after the agent performs action \(a\) in state \(s\). The update rule \"boot-straps\" old estimates of the q-values into new estimates. The training episode terminates when the positions of the agent and interest-point coincide. Figure 9 shows the outcome after 1000 training episodes. Figure 9: The single-agent, single stationary interest-point scenario. (**a**) The initial position of the agent is in cell (0,0). The interest-point is positioned in cell (2,2). (**b**) The number of training steps per episode converges to the optimum of 4. (**c**) The number of visits to each cell over the entire training process. Cells on the shortest path to the interest-point receive the most visits. (**d**) The estimated maximum q-value of each cell. Cells closer to the interest-point have more value. Figure 9b shows the number of steps per training episode converges approximately to 4 (which is the length of the shortest trajectory). This convergence is also evident in Figure 9c which shows the total number of visits to each cell over all training episodes. The starting point at position \((0,0)\) has the most visits. The intermediate positions on the shortest trajectories between \((0,0)\) and \((2,2)\) are also visible in Figure 9c. Positions outside these trajectories are visited rarely indicating that the \(\epsilon\)-greedy policy converges to the optimal one. Figure 9d shows the maximum q-value \(\max_{q}q(s,a)\) for each cell. The q-value peaks at position \((2,2)\) and a value of 500, because the agent at this position coincides with the interest-point. The surrounding cells include the expectation of this discounted future reward, yielding a pyramid shape. The \(\epsilon\)-greedy policy uses the q-values to determine the appropriate action in each state, choosing actions that maximize the expected return. This policy corresponds to the shortest path between the agent and interest-point. Now we add an additional stationary interest-point to the system and observe the effect on the learning process. The reward function with multiple interest-points is modified to assign \(r=500\) for every un-visited interest-point. The hyper-parameters and update rule are the same as before. Figure 10 shows the results after 1000 training episodes. Figure 10: The single-agent, two stationary interest-points scenario. (**a**) The initial position of the agent is in cell (0,0). The interest-points are positioned in cells (2,2) and (3,0) respectively.