# Strategies for Scaleable Communication and Coordination in Multi-Agent (UAV) Systems
## 3 Multi-Agent Planning and Control
### Hierarchical Reinforcement Learning  
The discussion thus far has focused on compressing the state-space, one of two factors driving complexity. The other factor is the size of the action-space, which also increases exponentially in the number of robots. If each robot has four possible actions (\"up\", \"down\", \"left\", and \"right\"), then the number of possible joint actions is \(4^{|V|}\). Hierarchical reinforcement learning (HRL) is a strategy for compressing action spaces [22]. The key idea in HRL is \"options\" (i.e., macro-actions) which specify sequences of primitive actions (\"up\", \"down\", \"left\", and \"right\") that take effect over single time-steps. Lower policies specify primitive actions, whereas higher policies specify options. The system policy is an appropriate fusion of these higher and lower policies. If the policy levels are independent, they can be trained separately. In reality, policy levels are inter-dependent and must be untangled in order for separate training to work properly. In multi-agent tracking, higher policies issue directives such as \"move from region A to region D\", as depicted in Figure 2(b). Options that execute this directive rely on sequences of lower policies such as \"move from region A to B\", \"move from region B to C\", \"move from region C to D\". Once the robot is in region D, the higher policy may issue new directives such as \"find interest-points within region D\", which correspond to different option sequences. There is a natural relationship between HRL and soft locally interactive structure in the state space; higher-order policies respond to aggregated (i.e., large-scale) phenomena of the kind that is not lost when the state-space is compressed. This relationship between action-space and state-space compression, deserves further attention in the literature. Option-spaces also grow exponentially with respect to the time-horizon window. For example, given the previous four primitive actions and a horizon of \(T\) time-steps, the number of possible options is \(4^{T}\). Some of these options are equivalent, in that they share the same start and end-points and have similar trajectories. To compress the option-space, each equivalence class can be reduced to an exemplar from the class. The problem is then choosing equivalence classes that don't sacrifice too much in the way of performance. This problem is largely application-specific. In multi-robot tracking, the ideal compressed option-space corresponds to a dynamic aerial transportation network, as depicted in Figure 4.  
Unmanned Aerial Traffic Management (UTM) systems, which allocate airspace to UAVs on-demand, are a current topic of interest in the aerospace community. There is clear overlap in scope between option-spaces in RL and airspace assignment which has yet to be fully explored. Figure 4: A self-adjusting options-induced skyway network. (**a**) The grid divides hierarchically into regions with the same number of interest-points at cells of the same level. (**b**) Options-induced trajectories move UAVs to different regions of each cell. Higher throughput trajectories occur at higher levels and are represented by thicker lines. Base-stations are denoted in blue. A major challenge in HRL is applying the ideas in Section 2 to each policy level of the option-space. Part of the problem is appropriately choosing the reward function for each sub-task, to achieve the desired net effect over all policy levels [23]. One family of strategies, based on the actor-critic approach, is the option-critic where each policy-level is assigned its own actor (estimate of the policy) and critic (value of the policy) [24]. There are numerous variations of this approach [25; 26; 27]. Q-Learning presents a different challenge rooted in the deadly triad of Section 2, where sampling from an experience-replay buffer makes successive updates look independent to the stochastic gradient descent algorithm.