ALMA: Hierarchical Learning for Composite Multi-Agent Tasks
Authors: Shariq Iqbal
Robby Costales
Fei Sha
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-agent
MARL
RL
reinforcement learning
HRL
hierarchical reinforcement learning

Abstract: Despite significant progress on multi-agent reinforcement learning (MARL) in recent years, coordination in complex domains remains a challenge. Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, we observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus on one subtask to the exclusion of all else in the environment. In these composite tasks, successful policies can often be decomposed into two levels of decision-making: agents are allocated to specific subtasks and each agent acts productively towards their assigned subtask alone. This decomposed decision making provides a strong structural inductive bias, significantly reduces agent observation spaces, and encourages subtask-specific policies to be reused and composed during training, as opposed to treating each new composition of subtasks as unique. We introduce ALMA, a general learning method for taking advantage of these structured tasks. ALMA simultaneously learns a high-level subtask allocation policy and low-level agent policies. We demonstrate that ALMA learns sophisticated coordination behavior in a number of challenging environments, outperforming strong baselines. ALMA's modularity also enables it to better generalize to new environment configurations. Finally, we find that while ALMA can integrate separately trained allocation and action policies, the best performance is obtained only by training all components jointly. Our code is available at https://github.com/shariqiqbal2810/ALMA

ALMA: Hierarchical Learning for Composite Multi-Agent Tasks
Shariq Iqbal'
Robby Costales University of Southern California
Fei Sha Google Research
Deepmind
Abstract
Despite sigmificant progress on multi-agent reinforcement learning (MARL) in recent years, coordination in complex domains remains challenge. Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus in the environment. In these composite tasks, successful policies can often be decomposed into levels of decision-making: agents are allocated specific subtasks and each agent acts productively towards their assigned subtask alone. This decomposed decision making provides a strong structural inductive bias, significantly reduces agent observation spaces, and encourages subtask-specife policies be reused and composed during training, as opposed to treating each new composition of subtasks as unique. We introduce ALMA, a general learning method for taking advantage of these structured tasks. ALMA simultaneously learns high-level subtask allocation policy and low-level agent policies. We demonstrate that ALMA learns sophisticated coordination behavior number of challenging environments, outperforming strong baselines. ALMA's modularity also enables it to better generalize to new environment configurations. Finally, we find that while ALMA can integrate separately trained allocation and action policies, the best performance obtained only by training all components jointly. Our code is available at http:///iittubbooo////aauuuaaaap 10AMLA
subtask to the exclusion of all else
Introduction
A certain set of real-world multi-agent tasks increase in complexity in proportion to the inter-agent coordination required rather than any inherent difficulty in the low-level skills. Take the example of firefighting (Figure 1): while coordinating firefighters across large city is more complex than in small one, the individual sub-task of putting out a fire remains similar across cities of sizes. Despite the recent success of cooperative multi-agent reinforcement learning (MARL) methods on wide range of domains [2, 16. 35, 5], they still struggle to learn sophisticated coordination behavior in sample-efficient manner on tasks of real-world complexity. And this is no surprise- not only do these tasks require that agents learn complex low-level skills execute sub-tasks, they must also incorporate coherent high-level strategies into their policies. Integrating both strategies and skills into the same action-level policy makes learning difficult.
Fortunately, for many relevant multi-agent tasks, while there may exist many different objectives that the agent population as whole must attend to, individual agents need only focus on isolated aspects of the environment at any given time. Consider again the setting of firefighting in city given a distributed set of resources. While there may be many fires occurring in the city at once, each firefighter can realistically only fight one at time. Thus, the optimal behavior in this setting can be
Work performed while at USC.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

expressed in terms of allocating firefighters to the right fires, and each firefighter effectively fighting the fire which they are assigned.
Suitable policies for this and many other real- world problems be formulated as bi-level decision-making processes [20]; agents are allo- cated the most relevant subtasks in the envi- ronment, then each agent selects actions with the purpose of completing its assigned subtask. We call settings that can be formulated in this way "composite tasks". Decomposing these tasks allows for more efficient learning for a num- ber of reasons. First, the learning process may benefit from the added structural inductive bias, alleviating the problem of incorporating both skills and strategy into the same low-level pol- icy. Second, by focusing solely on its assigned subtask, and ignoring the rest of the environ- ment, agent action-level policies can learn over significantly reduced spaee. Finally, tion, learning policies for individual subtasks and allowing the high-level controller to compose them may be simpler than learning single policy over all compositions of these subtasks.
Figure 1: The task of city-wide firefighting can be decomposed into allocating firefighters to the most urgent fires (top), and firefighters focusing solely on the fire they are assigned (bottom).
assume subtasks are drawn from the same distribu-
Our contributions are as follows. First, we introduce a general learning method, ALlocator-Actt Actor Multi-Agent Architecture (ALMA), for taking advantage of composite multi-agent tasks (84). ALMA learns a subtask allocation policy and low-level agent policies simultancously and designed in modular manner to promote the reuse of components across different compositions of subtasks. Learning the high-level controller poses challenges due to its large action space-at at any time step, the number of unique allocations of agents to subtasks is m". Our solution leverages recent methods [31] in learning value functions over massive action spaces and is designed modularly to handle variable quantities of agents and subtasks drawn from a shared distribution. While action-value function typically depend on the global state due to shared transition dynamics and reward functions that depend on all agents, we observe that for composite tasks we can decompose the environment into action-value functions to only depend on
independent sub-environments and redefine subtask- local information, improving their reusability and sample-efficiency. We next evaluate our proposed method on two challenging multi-agent domains ($5). Our complete method outperforms both state-of- the-art hierarchical and standard MARL methods well strong allocation heuristics. Furthermore, we evaluate the practicality of our subtask action-value function decomposition, demonstrating that learning fails without it while highlighting cases in which the required assumptions do not hold, Lastly, we show the importance of learning action policies in concert with allocation policies.
2 Related Work
Multi-Agent Task Allocation Task allocation is a long-studied problem in multi-agent systems and robotics [26, 9, 12, 8]. Given set of tasks and set of agents, the goal in this setting assign agents to tasks in a manner that maximizes total utility. A utility function which maps agent team-tast pairs to their utility is assumed to be given. Within the taxonomy of sub-classes this problem formalized by Gerkey and Matarié [9], our problem setting can be classified as ST-MR-IA IA (single-task agents, multi-agent tasks, instantaneous allocation). other words, agents cannot multi-task, tasks require the cooperation of several agents, and they are assigned instantaneously without incorporating any information regarding future task availability. We note that work in multi-agent task allocation often assumes the execution of tasks to be trivial; the focus is placed entirely on how to assign agents to a set of tasks in an efficient manner, and some knowledge of agents' effectiveness in each task is assumed. In our reinforcement learning setting, we only assume the ability to execute actions in an environment and observe global and subtask-speciffe rewards. Our focus is on enabling the application of MARL to complex composite tasks, rather than learning task allocation policies that compete with traditional approaches.
MDP Formulations for Multi-Agent Task Allocation Several works have proposed formulations of task allocation as an MDP, enabling the use of learning and planning-base approaches. Proper and Tadepalli [20] introduce the decomposition of the simultancous task setting into task allocation and

---

task execution levels. However, their allocation selection procedure does not consider how allocations may change in the future. Campbell et al. [6] address this shortcoming by taking future allocations into account when allocating agents to subtasks; however, they assume the pre-existence of low-level action policies. Notably, neither approach learns a tractable policy over the allocation space, and instead relies on exhaustive search or hand-crafted heuristics in order to select allocations.
Multi-Agent Reinforcement I. Learning Several recent works in MARL have addressed related problems to ours. Most relevantly, Carion et al. [7] introduce learning- basndd approach for task allocation that scales to complex tasks. Unlike our proposed setting, their work assumes pre-existing subtask execution policies, well as domain knowledge regarding how much an agent "contributes" to subtasks, which are used as constraints in a linear/quadratic program to prevent assigning more agents than necessary to subtasks. Our method, on the other hand, learns low-level policies, assumes less prior knowledge, and solves the problem of finding the best allocation by learning an allocation controller from experience rather than using an off-the-shelf solver with inputs generated by learned function and constraints from domain knowledge.
Shu and Tian [25] devise method whereby central "manager" agent provides incentives to self- interested "worker" agents to perform specific subtasks such that the manager's 's reward is maximized and incentives provided to workers
minimized. In this case, agents work on subtasks independently and are not cooperating towards a shared goal. Yang et al. [37] also consider a setting where multiple simultaneous tasks (described as goals) exist in shared environment; however, they assume that goals are pre-assigned to agents and therefore do not tackle the task allocation problem. Liu et al. [15] introduce COPA, a hierarchical MARL method which learns centralized controller in order to coordinate decentralized agents. Unlike our method, which utilizes task allocation for the purpose of decomposing complex settings to simplify learning, COPA's motivation is primarily alleviate the problem of partial observability during decentralized execution. We evaluate our method against COPA in order assess the effectivvness of our hierarchical decomposition in comparison more generic hierarchical MARL method with similar assumptions (i.c. no pre-trained policies, cooperative tasks, and no pre-existing task allocation).
3 Preliminaries
Task Framework Our setting builds on the general framework of Decentralized POMDPs [19] with entities [23, 10] by incorporating structure for multiple coexisting subtasks. Note that we differentiate between subtasks and tasks. We refer to independent jobs within environment as "subtasks" and the global collection of them (specified by the Dec-POMDP) as a task. This framework describes fully cooperative problems which can be specified by the tuple: (S, U, O, rr (r,, E I), A, μe, I). A is the set of agent entities. Each entity e has a state s We denote, for example, the state space of all agents as SA where {s A] SA- The state space of other sets of entities is denoted similarly. Each agent can execute actions w", and the joint action of all agents is denoted = {urla A} E U. Within the environment exist set of subtasks I where each is defined by a set of subtask-specific entities & and the subtask reward function r; Sz. x S x U + R. The relevant state (i.e. set of entity states) for subtask i is then denoted as s; = {s e E & A). We define& & as the set of all entities (including agents) in the environment: & (U,ez &,) U.A. The global state is the set := {s5 Ie E E) E S. Not all entities may be visible to each agent, so we define a binary observability mask: (($5,85 E {1,0). Thus, an agent's observation is defined as 0" [s"|pss s*) = 1,e E} O. P is the state transition distribution which defines the probability P(s'|s, u). Finally, we define the global reward function r S x U + R. In the simplest case, the global reward function can be sum of the individual subtask rewards; however, it can optionally encode global objectives (e.g. a scalar reward upon the completion of all subtasks).
Cooperative MARL via Value Function Factorization Cooperative MARL methods are con- cerndd with learning a set of policies which maximize the expected discounted sum of global rewards. Value function based methods do this by learning an approximation of the optimal Q-function:
nad (s. u) E Ee r(st, uz)
80 so=s, un=u, 8 P413a((-assuuu Ue41=arg max (8++1)))
= =r(s, u) + + E max (1o s/-
An optimal policy can be derived from this funnciio by taking the maximum valued action u from the current state; however, in the multi-agent setting it is often desirable for each agent to exccute their policy independently (i.e. in a decentralized fashion). In order to derive decentralized
3

---

policies, the Q-function is represented in manner that realizes the Aadvvddul--loobl-Maay (IGM)
principle [27] which states that each agent can greedily maximize their local utility Q" and maximize Qth as result. This has been accomplished through representing Q as summation of utilities [28], a monotonically increasing combination [21] of them, as well as several works which attempt to extend the representational capacity of the global Q-function to all functions satisfying the IGM principle [27, 36). The value function approximation is learned by training a neural network to minimize the following loss derived from the Bellman equation:
(0) := E2 [V (8( ( ((..M..r004)))))))
lot 1/2 Tr + 28 1L++1, arg max Q8" (8++1,
where 0 are the parameters of target network that is copied from @ periodically to improve sta- bility [17] and D is a replay buffer [14] that stores transitions collected by an exploratory policy (typically e-greedy). In cases of partial observability, the history of local actions and observations of) ) is used as proxy for the state 8; as input to the agents' utility functions Q".
Tp = (of. uf......ff
Hierarchical MARL Value-based single-agent Hierarchial RL methods [e.g. 13] typically learn a temporally abstracted high-level Q-function which selects goals upon which a low-level controller is conditioned. In adapting these methods for the multi multi-agent setting, one can choose to select goals in fully decentralized fashion [29] or through a centralized controller [15, 38, I]. Our work will consider the latter setting, where a single controller learns a value function over a shared goal space b E B:
Ni ((s.b) -nr rt + E [max Q(s' P. Tb]
(2)
This controller operates over a dilated time scale; one step from its perspective amounts to N₂ steps in the environment executed by low-level controllers conditioned on the goal. Tb. The low-level controllers are trained to maximize goal-specific rewards which can be manually defined or learned.
4 Allocator-Actor Multi-Agent Architecture
In the last section we introduced generic framework for value-based hierarchical MARL. Now we will present the details specific to our hierarchical MARL framework for composite tasks using subtask allocation, ALMA. In this case we define the action space of the high-level controller (whose value function is defined in Eqn. 2) as the set of all possible allocations of agents to subtasks. A subtask allocation refers to specific set of agent-subtask assignments, where each agent can only be assigned to one subtask at a time, and it is denoted by = {bo |a E A} E B, where ba E I represents the subtask that agent is assigned to. We then slightly abuse notation and denote the set of agents assigned to subtask as b, S. A and the set of all agents not assigned to task i as b where b, Ub = A. We then specify the joint action for agents assigned to subtask as up, = {u" E b Recall that the subtask-specific state s; includes the state of all agents. We denote the subtask-specific state including only the assigned agents as Sb,, = le E & U.
4.1 Subtask Allocation Controllers
One of the main challenges of learning Q-function for the high-level allocation controller is the massive action space. Our formulation of subtask allocation can be seen as a set partitioning problem, which is known to be NP-Hard [24, 9]. Q-Learning with the allocation action space requires finding the allocation with the highest Q-value at each step (Eqn. 2), as does deriving an action policy from this Q-function. At each step, we have choice of |4| possible unique allocations of agents to subtasks, and it is prohibitively expensive to evaluate each one and select the best. Van de Wiele al. [31] introduce "Amortized Q-Learning" which addresses the problem of massive action spaces in Deep Q-Learning by defining a "proposal distribution" which they train to maximize the density of actions with high values given a state. We adapt this idea for our allocation controller, where [bbs: s; Φ) is our proposal distribution over allocations. We can then sample from this distribution and select the allocation with the highest value, effectively approximating the maximization procedure required for Q-Learning, Our proposal distribution is learned with the following loss:
L(d; s) log f(b (s) Φ) AQ H-((-ss;;) ()).
(3)
where 6² b*(ss is the highest-valuee allocation from set of Np samples from the proposal distribution. Formally, 6" (ss) ;= arg max (or)(s) Q(s, b) where Bsamp (s) == {b....., b%, f()(=:))). We

---

then learn an approximation of Eq. 2 with the following loss, adapted from Eqn. 1:
Co(0) = 18 18 (1 Q Qo((
y/: = Σ retn Pe 6' (8L+NN))
(4)
where transitions (st, bi, En Tstn St+N,) ) are sampled from replay buffer. Learning a proposal distribution and value function over a combinatorial space closely parallels recent work in learned combinatorial optimization [3, 11, 4].
We construct our proposal distribution modularly that each module can leverage the fact that subtasks are drawn from a shared distribution. These modules consist of an agent embedding function fhh Sa - RRd, subtask emhedding function f Sg, IRd and subtask embedding update function f R + R which is used to update the embedding of the subtask that an agent was assigned to. We first construct f in a factorized ruto-regressive form: f(bls) = IIae f [s, b b"*), where b <a are the allocations made prior to assigning agent a. Thus, we are iteratively assigning agents to subtasks while accounting for allocations already determined.
Since the number of subtasks present at any time is variable (e.g. some subtasks may be finished before others), we construct each auto-regressive factor as composition of outputs generated by the modules described above, in fashion similar to pointer-networks [34]. Pointer networks are commonly used in RL applications to handle variable-sized action spaces where the individual actions correspond to entities/groups [35]. We first generate set of subtask embeddings g: = (8g, encoding the state of the non-agent entities belonging to each subtask, as well as embeddings describing each agent's state h. = f (s*).. Logits are computed as the dot-product between agent embeddings and subtask embeddings: [bb b <a) = s (g ha). We then sample b ~ f. and update the selected subtask's embedding as g gb> + f (gb-, ha) such that other agents' allocations can take into account existing ones.
4.2 Subtask Execution Controllers
Given subtask allocation b, we must learn low-level action- value functions as described in Section 3. Specifically, for each team by we learn value function based on the subtask rewards rú. Recall that the subtask reward function depends on all agents' actions and states since any agent can potentially contribute to any subtask. From the perspective of the team assigned to subtask i, we treat the other subtasks' agents as part of the environment by taking the expectation over actions sampled from their optimal policies:
(s, Ub.; b) = E.
-E (s; u) + t max (s" :b]
us, 3 2'-((: a,
(5)
While this function depends on the full global state since all entities can feasibly influence all other entities" state transitions and any agent can potentially contribute to the rewards of any subtask, in composite tasks the set of truly relevant information may be much smaller for optimal policies. Reconsider the example from Sec. where firefighters are agents and buildings are subtasks. While is possible for any firefighter to put out any fire, given optimal policies we expect firefighters to only put out the fire they are assigned to. Moreover, in the optimal setting, firefighters will not be directly interacting with firefighters at other buildings. As such, for composite tasks, we can decompose the environment into a set of independent sub-environments and rewrite Eqn. as follows:
Q10 ub, b) r. (sb., ub.) + EB mmax (sb,, : b) (/pl, (Ple,, un,))
(6)
where ph is the transition distribution of sub-environment and the transitions of the full environment can be written in the following factored form: P(s'|s, u) Iler Ph Ph |sb. us )ss, s' E S Given subtask allocation b, S is the set of global states visited by the optimal subtask policies, mi. Finally, r is the sub-environment reward function that depends only on local information defined as: Br? Se, x So. x Ub. + R s.t. (sb.: ub.) = (s, u)yss E S u E U.
Note that the Q-function now only depends on "local" state, and, as such, can be approximated more easily due to the smaller function input space. Moreover, the case where subtasks are drawn from the same distribution, Q-functions defined over the local state will more easily generalize to other instantiations of the same subtask since they no longer depend on the context outside of that subtask's 'ss state. We validate the usefulness of this decomposition empirically in Section 5.
5

---

&
=
}
0j E
b. = {a
8 =
8a
bb. = E
Bb;
TT. (- s6,
2bb
C2
11
C
S
II(
b
Subrude
aby {8 8 ,0000 ,00
by
1/ ~ 10th ( 18by
BS
Subtask Allocation
Subtaskk Exacution
Figure 2: ALMA computing agent actions (specifically agent as) given the current state. Subtask allocations b are updated by centralized controller II every N, steps, and then agent policies π select low-level actions u a decentralized fashion given their local state.
Learning a function that approximates Eqn. requires predicting Q-values for teams of varying
sizes, as the quantity of agents assigned to a subtask is not fixed. In fact, each unique combination of agents assigned subtask i can be seen as unique Dec-POMDP. We rely on recent work [10] which learns factorized multi-task Q-functions for teams of varying sizes by sharing parameters via attention mechanisms. This work falls under the category of factorized value function methods for cooperative MARL described in $3, and as such, represent each subtask-specific Q-function as monotonic mixture agent utility functions computed in a decentralized fashion, such that agents can act independently without communication.
4.3 Training and Execution Details
We provide an overview of the complete subtask allocation and execution procedure in Figure 2. Subtask allocations b are selected in a centralized fashion every N, steps which then determines the set of low level policies to execute. We define the high-level policy II(b) s) == 1(b (s)) where bz (s) is the highest valued allocation sampled from the proposal distribution, as defined in Section 4.1. The per-agent low-level policy for agent assigned subtask defined as r"("" $b, b) == lIu" = arg max,, .r Q" (sb., u b)) where Oi is the agent's utility function which is monotonically mixed with all b, to form !!
When collecting data during training, we select random low-level actions with probability c in order to promote exploration. We use types of exploration in the allocation controller. With probability c" we select a full allocation sampled from the proposal distribution, rather than taking the highest valued one. Then, with probability e", we randomly select subtask allocations independently on a per-agent basis. All exploration probabilities are annealed over the course of training, and the annealing schedules (along with all hyperparameters) are provided in Appendix D.
To handle state spaces of variable size (i.c. variable quantity of entities), we use attention models [33] as in [10] for all components. Network architectures provided in Appendix A. In order to achieve the partial views over the global state required by our redefined low-level Q Q-function in Eqn. 6, we masking in attention models to prevent agents from seeing certain entities.
5
Experiments
We evaluate ALMA on two challenging environments, described below (more in Appendix B).
5.1 Environments
SAVETHECITy
Inspired by the classical Dec-POMDP task "Factored Firefighting" [18], and based on the scenario described in Sec. 1, we introduce SAVETHECITY, an environment where agents must coordinate to extinguish fires and rebuild a burning city (see Fig. 3). In this version, agents are capable of contributing to any subtask, must physically navigate between buildings through low-level actions, and there are different agent types each with distinct abilities that are crucial for success. The subtask allocation function must discover these abilities from experience and use this knowledge distribute agents effectively. The task also shares similarities with the Search-and -aaec--and-Rescue task from [15], but we not consider partial observability and we introduce agents with diverse capabilities.
STARCRAFT Our next environment the StarCraft multi-agent challenge (SMAC) [22] which involves a set of agents engaging in a battle with enemy units in the StarCraft II video game. We specifically consider the multi-task setting presented by Iqbal et al. [10] where each task presents unique combination and quantity of agent and enemy types. As such, agents cannot learn fixed

---

64
ALMA
6.5
11
rurl
How (1rs)
64
COOS
63
Fatt
6.5
000
Sup
SHp
Step
Figure 4: STARCRAFT II training curves, left to right: S&7. (a) disadvantage and (b) symmetrie; MMM (c) disadvantage and (d) symmetric. Shaded region standard deviation across 5 seeds.
strategies, and must instead adapt strategies to their set of teammates and enemies. While the original setting consists of single enemy army attacking the agents, inspired by the full game of StarCraft we increase the complexity by introducing multiple enemy armies (see Fig. 5) which periodically attack and retreat independently and the additional objective of defending base. By doing do, we essentially composing several tasks of similar complexity to those considered by the current literature and requiring agents to learn not only how solve those tasks but also how to distribute agents across tasks in manner that is globally optimal. Within this setting, we consider four different variations of army compositions based on those presented in [10]. In one set of settings we give agents and enemies a symmetric set of units, while in the other, agents have one fewer unit than the enemies, Within each set we consider two different pools of unit types: "Stalkers and Zealots" (S&Z. and "Marines, Marauders, and Medivacs" (MMM). The former includes mixture of melee and ranged units, while the latter includes units that are capable of healing.
5.2 Results
In our experimental validation aim to an- swer the following questions: 1) Learning Effi- cacy: Is ALMA effective in improving learn- ing efficiency and asymptotic performance in comparison to state-of-the- hierarchical and non-hierarchical MARL methods? 2) Allocation Strategic Complexity: Are the learned alloca- tion policies non-trivial or are any benefits of ALMA purely gained from the subtask decom- position? 3) Decomposition Validity: What, if anything, do we gain from the sub-enviromment and how does ALMA fare when the required assumptions are broken? 4) Joint Training: Do we receive any benefits from training allocation and execution controllers jointly?
0.6
Method
No mark
(A)
Succes Rate
0.4
Heariele
Flat
0.2
(b)
0.0
0.00
0.25
0.50
0.75
0.00
Stop
Figure 3: SAVETHBBCITY example task and train- ing curves. Shaded region is a standard deviation across 8 seeds.
Learning Efficacy First, aim evaluate the effectiveness of our hicrarchical abstraction in comparison to state-of-the-as cooperative MARL methods in variable entity multi-task settings. We compare to two non-hierarchica multi-agent baselines: QMIX [21] augmented with self-attention [33], to extract information from variable quantities entities (A-QMIX), and REFIL [10], state- of-the-art art approach built top of A-QMIX for generalizing across tasks with different compositions of agent and entity types. Both methods are referred to as "Flat" in our figures and differ depending on the environment. REFIL is used for STARCRAFT tasks, but we find that it does significantly improve on A-QMIX in S AAVETHECITY So we use A-QMIX in that setting. Evaluating these methods isolate the effectiveness of our hierarchical decomposition,
on tasks allows same algorithms to train low-level controllers in ALMA.
the
Next, compare state-of-ther approach in hierarchical MARL for varying entity settings: COPA [15]. COPA makes the assumption, similar to ALMA, communicate with decentralized agents periodically. We ensure that the communication frequency of COPA matches that of our method and that it is provided subtask labels for entities, so it receives a similar amount of information, even though it does not explicitly leverage the subtask decomposition.
centralized agent that is able to
In both SAVETEECITY and STARCRAFT, we find that ALMA is able to outperform all baselines in most settings. Both A-QMIX (Flat) and COPA are unable to converge the allotted timesteps in SAVETHECITy (Fig. 3), and REFIL (Flat) and COPA are only competitive with ALMA in terms of both sample efficiency and asymptotic performance in one setting: MMM
reasonable policy within
7

---

1 15
E= 40
45
(e)
Legend
Agei Every
@
Figure 5: Walk-through learned ALMA policy on a sample STARCRAFT task. Agents start (t = 0) at the center, and two enemy armies immediately begin to attack (a). ALMA learns there is an advantage in numbers (e.g. "focus firing" on single enemies) and allocates most agents to Subtask 1 (b). One Zealot is additionally assigned to each attacking army prevent the base from being stormed (c-d). This strategy the very first time-step (a). Around
determined from 40, a Stalker reallocated to Subtask 3 (e), as the Zealot previously assigned was defeated, leaving a vulnerability, and Subtask has almost heen completed by the other agents. Once Subtask 1 is complete (t = 45), most agents are allocated to Subtask 3, the largest army, but one Stalker (f) is allocated to assist the Zcalot at Subtask 2 (g) since it is low on health. Once Subtask 2 is complete, the remaining agents allocated Subtask 3 complete the task.
Symmetric (Fig. 4d). These results highlight the importance of ALMA's hicrarchical abstraction as well as the sub-environment decomposition used in order to accelerate learning of low-level policies.
Interestingly ALMA appears supply the greatest performance gains the environments that most difficult, as judged by the absolute success rates. ALMA stands out in S&Z disadvantage (Fig. 4), where the next best methods achieve 20% success, while it only roughly matches the performance of the top performing methods in MMM symmetric where the highest success rates are around 75 We hypothesize that when agents are faced with such disadvantage, coordinated strategy becomes more crucial.
Allocation Strategic Complexity Next, hope to learn whether our allocation controller is learning complex non-trivial strategies. As such, we implement allocation heuristics derived from domain knowledge in each setting and only learn the low-level controllers. These methods serve as strong baseline for learned allocation, as they simplify the learning problem for low-level controllers by leveraging the sub-environment decomposition (i.e. masking subtask-irrelevan entities) and having fixed high-level allocation strategy. For SAEETHECITY, the heuristic allocates each agent to the nearest building at which they are most useful according their capabilities. While relatively simple, devising more sophisticated allocation strategy is nontrivial as there are many factors to weigh. For STARCRAFT, we devise two heuristics: the first (matching) allocates agents to enemy armies by matching unit types such that each individual battle is fair. The other (dynamic) only considers enemy armies that are currently attacking and also attempts to match the unit composition.
In SAVETHECITY (Fig. 3) we find the heuristic is able to learn quickly at the beginning as the action-level policies improve, but it converges lower than ALMA. In STARCRAFT (Fig. 4) we find that at least one of the heuristics can be competitive in some cases (e.g. S&Z Symmetric). although which heuristic performs best is not consistent across settings, and ALMM always outperforms With these results we can conclude decomposing the task into subtasks to be solved by simpler task execution controllers is not sufficient for superior performance our environments and sophisticated high-level strategy must be learned.
In Fig. 5 qualitatively demonstrate some of the strategies that ALMA is able to learn. We find that ALMA is able to allocate agents to subtasks in a manner that considers long term consequences and balances priorities gracefully (e.g. defending the base defeating enemies).
Decompasition Validity In order to validate the sub-environment decomposition introduced in Eqn. we ablate our approach to exclude the task-specific masking performed on each agent's observation. This method is referred to as "No mask" in Figures 3 and 4, where we find that our method experiences a significant deterioration in performance when not masking irrelevant information.

---

To test the boundaries of the required assump- tions (independent transitions), we evaluate on settings in STARCRAFT which violate them. While we train the agents only in cases where en- emy armies are maximally spread out from each other around the agents' base, in our evaluation setting we allow for armies spawn arbitrarily close to one another. In this case, for example, agents assigned to one army may be attacked by other armies or bump into agents attacking other armies, violating subtask transition inde- pendence. We plot the performance of ALMA alongside REFIL (Flat) in Fig. 6a, where the distance between armies varies along the x-axis. The further right, the more similar the tasks are to what is seen during training.
Succen Ibte
0.5
16
Mac
AIMA AMMA IN
Erory Arto Sprcal
Figure 6: Further analysis in STARCRAFT S&Z disadvantage setting. Left (a): Varying spread of enemy armies. Right (b): Zero-shot evaluation with alternate action policies.
While REFIL's performance deteriorates smoothly as the composition of subtasks become more dissimilar to those seen during training, ALMA maintains steady performance and even sees slight uptick performance towards the middle. While surprising, this difference in generalization capability be attributed to the modularity of approach. While REFIL treats each unique composition of subtasks as novel, the modules comprising our approach (specifically the subtask- execution controllers and allocation proposal network) only see information relevant to their subtask, which are individually drawn from the same distribution as seen during training. ALMA' 's increase in performance in the middle easier for agents to switch between tasks. Ultimately, as armies become closer together, our assumptions begin to fail, subtasks blur together, and we finally see ALMA's 's performance drop off.
be attributed the fact that armies being closer together makes
Joint Training Although ALMA is able to learn all components (subtask allocation and execu- tion/action controllers) jointly, we evaluate its ability to utilize agent policies trained by a different method in zero-shot manner. In particular, we what happens when replace A ALMA' action- level policies with those trained with a heuristic. We note that the procedure for training the low-level controllers is identical between these methods and the only difference the allocation strategy used throughout training. We see in Figure 6b that without further training. ALMA'S 's superior allocation strategy able to quadruple the effectiveness of the policies learned with the heuristic. Ultimately though, the best performance is only attainable through training allocation and execution controllers jointly, the execution controllers can learn to succeed in the settings the allocation controllers put them in, and vice versa, creating a virtuous feedback loop.
6 Conclusion and Future Work
In this work have introduced ALMA, general learning method for tackling variety of multi- agent composite tasks in which each subtasks' rewards and transitions can be assumed as independent. By simultaneously learning a high-level allocation policy and action-level agent policies, ALMA is able to succeed in settings which are difficult for flat methods, and in which well-performing heuristies are difficult come by. One promising direction for future work is to equip A ALMA'S allocation policy with more sophisticated exploration methods. Although the stochasticity resultant from the e-greedy procedure and the proposal distribution sampling yields surprisingly good results, more intelligent exploration procedure could even better alleviate the challenges inherent to learning over such large combinatorial action space. We expect such an improvement would, for example, prevent some of the variance we observe in SAVETHECITY (Figure 3). Another direction for future work is discover subtasks and/or observation masks from the environment automatically. We believe this work has immediate practical implications in real-world settings that can be described as composite tasks, such as warehouse robotics.
7 Acknowledgements
We thank Sébastien Arnold and the anonymous reviewers for their feedback. This work is partially supported by NSF Awards 11S-1513366/ 1632803/1833137,, CCF-1139144 DARPA Award#: FA8750- 18-2-0117, 17, gifts from Facebook and Netflix, and ARO# W9 1N1-NF21 1-0241 and W911NFF15-1-0488
DARPAD3M Award UCB-00009528, Google Research Awards,

---

References
[1] Sanjeevan Ahilan and Peter Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. arXiv preprint arXiv: arXiv: 1901.0849, 2019.
[2] Trapit Bansal, Jakub Pachocki, Szymon Sidor, llya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. International Conference on Learning Representa- lions, 2018.
[3] Irwan Bello, Hieu Pham, Quoe V Le, Mohammad Norouzi, and Samy Bengio. Neural combina- torial optimization with reinforcement learning. arXiv preprint arXiv:1611.09944 2016.
[4] Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d' h'rooizon. European Journal of Operational Research, 2020.
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.
2019.
[6] Trevor Campbell, Luke Johnson, and Jonathan P How. Multiagent allocation of markov decision process tasks. In 2013 American Control Conference, pages 2356-2361 IEEE, 2013.
[7] Nicolas Carion, Nicolas Usunier, Gabriel Synnaeve, and Alessandro Lazaric. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning. In Advances in Neural Information P Processing Systems, pages 8128-8138, 2019.
[8] Ali Dorri, Salil S Kanhere, and Raja Jurdak. Multi-agent systems: A survey. leee Access, 6: 28573-28593, 2018.
[9] Brian P Gerkey and Maja J Matarié. A formal analysis and taxonomy of task allocation in multi-robot systems. The International journal of robotics research, 23(9):939-954, 2004.
[10] Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4596-4606, PMLR, 2021.
[11] Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[12] Alaa Khamis, Ahmed Hussein, and Ahmed Elmogy. Multi-robot task allocation: A review of the state-of-the-art. Cooperative Robots and Sensor Networks 2015, pages 21-51,2015,
[13] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforeement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 00:3675-36833 2016.
[14] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning ((3):293-322 1992.
[15) Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach- player multi-agent reinforcement learning for dynamic team composition. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6860-6870, PMLR, 18-24 Jul 2021.
[16] Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Eslami, Daniel Hennes, Wojciech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105.12 12196. 2021.
[17] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
10

---

[18] Frans A Olichock, Matthijs TJ Spaan, Nikos Vlassis, and Shimon Whiteson. Exploiting locality of interaction in factored dec-pomdps. In Int. Joint Conf. on Autonomous Agents and Multi Agent-Agent Systems, pages 517-524, 2008.
[19] Frans Olichoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[20] Scott Proper and Prasad Tadepalli. Solving multiagent assignment markov decision processes. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume I. pages 681-688. Citeseer, 2009.
[21] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning, In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4295-4304, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018.
[22] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, International Foundation for Autonomous Agents and Multiagent Systems. 2019.
[23] Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. In Advances in Neural Information Processing Systems 32, pages 9927-9939. Curran Associates, Inc., 2019.
[24] Onn Shehory and Sarit Kraus. Methods for task allocation via agent coalition formation. Artificial intelligence, 101(1-2):155-000, 1998.
[25] Tianmin Shu and Yuandong Tian. M3rl: Mind-aware multi-agent management reinforcement learning. In International Conference on Learning Representations, 2019.
[26] Reid G Smith. The contract net protocol: High-level communication and control in distributed problem solver. IEEE Transactions on computers, (9(12):110441111) 1980.
[27] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference Machine Learning, pages 5887-5896, 2019.
[28] Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decoomosittoon networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference Autonomous Agents and MulriAgent Systems, AAMAS 18, pages 2085 2085,2087, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
[29] Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Zhaopeng Meng, Changjie Fan, et al. Hierarchical deep multiagent reinforcement learning with temporal abstraction. arXiv preprint arXiv:1809.09332. 2018.
[30] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-mmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2): 26-31, 2012.
[31] Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning enor- mous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116. 2020.
[32] Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih and David Silver. Learning values across many orders of magnitude. Advances in Neural Information Processing Systems, 29:4287-4295, 2016.
11

---

[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, pages 6000-6010, 2017.
[34] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in Neural Information Processing Systems, 28:2692-2700, 2015.
[35] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-333 2019.
[36] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-learning International Conference on Learning Representations, 2021.
[37] Jiachen Yang. Alireza Nakhaei, David Isele, Kikuo Fujimura, and Hongyuan Zha. Cm3: Coop- erative multi-goal multi-stage multi-agent reinforcement learning. In International Conference on Learning Representations, 2019.
[38] Jiachen Yang. Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforcement learning with skill discovery. In International Conference on Autonomous Agents and Multi Auent Systems, 2020.
12


Reviews:
[{"review_id": "xIem61namDz", "reviewer": "Reviewer_tJvo", "summary": "The paper presents a method for achieving hierarchical multi-agent learning. The proposal network is trained to assign subtasks to each agent and each agent will be assigned to accomplish its own task. The proposed method is evaluated on the save the city tasks and starcraft II multi-agent micro-management tasks. It shows better performance on the benchmarks. ", "questions": "Please see the weakness above. ", "limitations": "I didn't see some potential negative social impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The paper presents an interesting idea of hierarchical multi-agent learning using the pointer network. The task assignment and skill learning in the multi-agent setting is a good approach and can be applied to many real-world tasks.\n\n2. The writing is clear and the formulation is straightforward. \n\nWeakness: \n1. My main concern about the paper is lacking baselines. I think in order to show the superior performance of hierarchical multi-agent learning, we need to compare it against the simple centralized training and decentralized execution method. I didn't see the comparison in the Starcraft setting against papers like QMIX, RODE, ROMA. I think will be convincing if the paper compares the proposed algorithm with these methods, in order to show the benefit of hierarchical learning. \n\n2. I am a little confused about the performance in the Starcraft setting since the results in QMIX, RODE and ROMA are much better than the presented results in FIg. 4. I am wondering if there is some difference in the setting, or is it a fair comparison?\n\n3. I am very interested to see some ablation studies, like the entropy coefficient in Eq. 3 and how the frequency of assigning subtasks to the agent will affect the performance. In particular, if the agent doesn't accomplish the assigned task when the high-level controller decides to change the task, what will happen?\n\n4. It's a minor issue but I feel it will make the paper more convincing if we can test more tasks on the Starcraft settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "072pZoGB1bW", "reviewer": "Reviewer_JvaG", "summary": "This paper presents a method of hierarchical multi-agent reinforcement learning, specifically for the case where there is a high-level centralized allocator that assigns low-level subtasks to different agents. It assumes that the environment POMDP has some predefined subtasks, from which we can learn low-level policies, and additionally learn the high-level allocator policy simultaneously. The main challenges include handling variable state and action spaces, as well as handling a very large action space for the allocator policy which are addressed via attention models (state) and pointer networks (action). The paper runs experiments on SaveTheCity and StarCraft, two challenging multi-agent environments that involve different number of agents in different configurations. Finally the paper speculates that this formulation may have applications in warehouse robotics.", "questions": "L83: \"We note that work in multi-agent task allocation often assumes the execution of tasks to be trivial\"\nL95: \"however, they assume the pre-existence of low-level action policies\"\n\nSeveral points were made regarding the difference in ALMA being that it learns low-level action policies as well. But, it seems that we are assuming the POMDP itself has predefined subtasks with its own rewards. To me, this assumption makes learning the low-level action policies trivial, since we can just run any RL algorithm on the subtasks. Is my understanding correct?", "limitations": ".", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-executed, in the sense that it identifies practical challenges with the allocator-actor framework and addresses them in very sensible ways. For example, dealing with large action spaces using Amortized Q-Learning and proposal networks, using pointer networks and attention models.\n\n- The experiments are done on challenging environments, and were tweaked to be even slightly more challenging (introducing more enemies in StarCraft), which is impressive. The results are convincing too.\n\n- The writing is exceptionally clear. It gives enough clarification on background techniques and leaves enough room to describe ALMA in detail. Of particular note are the wonderful figures (Figs 2, 3), which are not just helpful/informative, but are truly some of the prettiest I've ever seen.\n\n**Weaknesses**\n\nMy main reservation about this paper is the motivation of the problem formulation. There are many, many different setups in hierarchical MARL, each with different assumptions on the environment, observability, centralized/decentralized, etc.... While the Related Work section was helpful, I didn't feel that it was comprehensive enough to really convince me of the importance this allocator/actor framework. Doing a better job in this section would prevent this from feeling like 'yet another MARL framework'.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-br6ZuDmugH", "reviewer": "Reviewer_w9pc", "summary": "This paper proposes a general learning method for solving composite tasks in multi-agent RL settings. They propose to learn a high-level allocator with a low-level  action executor. To alleviate problems such as large action spaces, the authors introduce various techniques to make the learning possible. Finally, the authors applied ALMA to a firefighting problems and the SMAC benchmarks.", "questions": "Can you discuss more in depth about when your allocator is better than a method with strong heuristics?\n ", "limitations": "We appreciate the author for being upfront. The limitations are 1) the exploration are simple e-greedy; 2) observation masks can be discovered automatically. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written.\n2. The problem is interesting. RL has shown great progress as a policy optimization tool. However, in many real-world applications, the structure of the tasks should be considered. \n3. The performance is better than baseline on various benchmarks. \n\nweaknesses:\n1. The generality of ALMA is not convincing. For different tasks, the task allocator might have different action spaces. In some certain problems, the scheduling problem can be solved by other non-deep methods. The authors only empirically show  that the proposed method is better than others without sufficient discussion.\n2. The masking trick seems important. But this is pure inductive bias.\n3. The testing environment are on the simple end of the spectrum. It would be nice to have a robot task in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept