Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
Authors: Shariq Iqbal
Christian Schroeder de Witt
Bei Peng
Wendelin Boehmer
Shimon Whiteson
Fei Sha
Venue:  (2021)
Keywords: MARL
multi-agent reinforcement learning
value function factorization
attention

Abstract: Real world multi-agent tasks often involve varying types and quantities of agents and non-agent entities; however, agents within these tasks rarely need to consider all others at all times in order to act effectively. Factored value function approaches have historically leveraged such independences to improve learning efficiency, but these approaches typically rely on domain knowledge to select fixed subsets of state features to include in each factor. We propose to utilize value function factoring with random subsets of entities in each factor as an auxiliary objective in order to disentangle value predictions from irrelevant entities. This factoring approach is instantiated through a simple attention mechanism masking procedure. We hypothesize that such an approach helps agents learn more effectively in multi-agent settings by discovering common trajectories across episodes within sub-groups of agents/entities. Our approach, Randomized Entity-wise Factorization for Imagined Learning (REFIL), outperforms all strong baselines by a significant margin in challenging StarCraft micromanagement tasks.

Under review as a conference paper at ICLR 2021
RANDOMIZED ENTITY-WISE FACTORIZATION FOR REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Real world multi-agent tasks often involve varying types and quantities of agents and non-agent entities; however, agents within these tasks rarely need con- sider all others at all times in order to act effectively. Factored value function ap- proaches have historically leveraged such independences to improve learning ef- ficiency, but these approaches typically rely on domain knowledge to select fixed subsets of state features to include in each factor. We propose to utilize value function factoring with random subsets of entities in each factor as an auxiliary objective in order disentangle value predictions from irrelevant entities. This factoring approach is instantiated through simple attention mechanism masking procedure. We hypothesize that such an approach helps agents learn more effec- tively in multi-agent settings by discovering common trajectories across episodes within sub-groups of agents/entities. Our approach, Randomized Entity-wist Factorization for Imagined Learning (REFIL), outperforms all strong baselines by a significant margin challenging StarCraft micromanagement tasks.
INTRODUCTION
Many real-world multi-agent tasks contain scenarios in which an agent must deal with varying numbers and/or types of cooperative agents, antagonist enemies or other entities. Agents, however, can often select their optimal actions while ignoring subset of agents/entities. For ex- ample, in the sport of soccer, a "breakaway" occurs when an attacker with the ball passes the defense and only needs to beat the goalkeeper in order to score (see Figure [ In this situation, only the opposing goalkeeper imme- diately relevant to the attacker's success, the attacker can safely ignore players other than the goalkeeper for the time being. By ignoring irrelevant context, the attacker can generalize this experience better to its next break- away. Furthermore, soccer takes many forms, from casual 5 vs. 5 to full scale 11 vs. matches, and breakaways oc- cur in all. If agents can identify independent patterns of behavior such as breakaways, they should be able to learn more efficiently as well as share their experiences across all forms of soccer.
Figure 1: Breakaway sub-scenario in soc- cer. Agents in the yellow square can ignore the context outside this region and still predict their success effectively.
Value function factoring approaches attempt to leverage independences between agents, such as those our soccer example, by learning value functions as combination of independent factors that depend on disjunct subsets of the state and action spaces (Koller & Parr 1999). These subsets are typically fixed in advance using domain knowledge about problem at hand, and thus are not scalable to complex domains where dependencies are unknown and may shift over time. Recent approaches in cooperative deep multi-agent reinforcement learning (MARL) factor value functions into separate components for each agent's action and observation space in order to enable decentral- ized execution (e.g., VDN (Sunehag et al. 2018). QMIX (Rashid et al. 2018)). These approaches leam utility function for each agent that only depends on the agent's vations. The global Qavalue is then predicted as some monotonic combination these utilities in order allow agents greedily select their actions with local information while maximizing the
action and its obser-

---

Under review as a conference paper at ICLR 2021
global Q. These approaches are able to effectively leverage independence between agents' local actions and observations, however, we note that observable entities are provided by the environment and are not all necessarily relevant to agent's is value function.
We build on these recent approaches by additionally factoring the observation space of each agent into factors for sub-groups of observed entities. Unlike classic works which factor the state or ob- servation spaces, our work does not depend on fixed subsets of features designated through domain knowledge. Instead, we propose randomly select sub-groups of observed entities and "imagine" the predicted utilities within these groups for each agent. These terms will not account for potential interactions outside of the groups, so we include additional factors that estimate the effect of the entities outside of each combine all factors using mixing network (as in QMIX. Rashid et al. 2018), which allows our model to weight factors based on the full state context. We hypothesize this approach is beneficial for two reasons: 1) randomly partitioning entities and predicting returns from disjunct factors allows our model to explore all possible independence relationships among agents and entities, teaching agents to ignore irrelevant context when possible and 2) by teaching our models when they can ignore irrelevant context, they will learn more efficiently across varied settings that share common patterns of behavior, such as breakaways in soccer. The loss for training randomized factorization added to the QMIX loss (i.e., using full observations) as auxiliary objective. Our reasoning is again twofold: 1) we must learn the true returns to use as a target prediction for a Q-learning loss. 2) we do not know a priori which entities are unnecessary and thus need to learn policies that act on full observations.
on each agent's 's utility. In order to estimate the true returns, we
Our entety-wise factoring procedure can be implemented easily in practice by using simple mask- ing procedure in attention-based models. Furthermore, by leveraging attention models, we can apply our approach to domains with varying entity quantities. Just as soccer agent experiencing break- away can generalize their behavior across settings (5 vs. 5, 11 vs. 11, etc.) if they ignore irrelevant context, we hypothesize that our approach will improve performance across settings with variable agent and entity configurations. We propose Randomized Entity-wise Factorization for Imagined Learning (REFIL) and test on complex StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) tasks with varying agent types and quantities, finding it attains improved performance over state-of-the-aae methods.
2 BACKGROUND AND PRELIMINARIES
In this work, we consider the decentralized partially observable Markov decision process (Dec- POMDP) (Oliehoek et al. 2016), which describes fully cooperative multi-agent tasks. Specifically, we utilize the setting of Dec-POMDPs with entities (Schroeder de Witt et al. 2019).
Dee-POMDPs with Entities are described as tuples: (S, U, P, entities in the environment. Each entity has a state representation sf, and the global state is the set s = [s E E} E S. Some entities can be agents a E A S Σ. Non-agent entities are parts of the environment that are not controlled by learning policies (e.g.. landmarks, obstacles, agents with fixed behavior). The state features of each entity comprise of two parts: 8e = [ Φ*) where Ie represents the description of entity's current state (e.g., position, orientation, velocity, etc.) while 6- E & represents the entity's type (c.g., outfield player, goalkeeper, etc.), of which there are a discrete set. An entity's type affects the state dynamics importantly, it remains fixed for the duration of the entity's 's existence. Not all entities may be visible to each agent, so we define a binary observability mask: p(s", ((s"",s") s²) E (1,0), where agents can always observe themselves jus"" s²) = 1, Va E A. Thus, an agent's 's observation is defined as 0a = [s"] 55) = 1,e E E E O. Each agent a can execute actions u", and the joint action of all agents is denoted as
μ)). E is the set of
well as the reward function and,
(a E A} E U. P is the state transition function which defines the probability P(s'|s |s. u). r(s, u) is the reward function which maps the global state and joint actions to single scalar reward.
We do not consider entities being added during an episode, but they may become inactive (c.g., a unit dying in StarCraft) in which case they no longer affect transitions and rewards. Since s and u are sets, their ordering does not matter, and our modeling construct should account for this (e.g.. by modeling with permutation invariance/equivariance (Lee et al. 2019). In many domains, the set of entity types present { E & is fixed across episodes. We are particularly interested in cases where quantity and types of entities are varied between episodes, identifying independence relationships between entities is crucial to generalizing experience effectively in these cases.
2

---

Under review as a conference paper at ICLR 2021
Learning for Dec-POMDPss We aim to learn set of policies that maximize expected discounted reward (retums) in some MDP. Q-learning is specifically concerned with learning an accurate action- value function Q (defined below), and using this function to select the actions that maximize expected returns. The optimal Q-function for the Dec-POMDP setting is defined as:
(s u) = E ΣE r(s,, ut)
By=s, un=u, 8t-1 st,ue)
Mel1 marg max (*
r(s, u) + E max ()) |/
(1)
Partial observability is typically handled by using the history of actions and observations as a proxy for state, typically processed by recurrent neural network (RNN, Hausknecht & Stone 2015: Q (Te, u.) x (st, where the trajectory (i.e., action observation history) is T? (%) 20- ... of) and T. = JaEA-
Work in deep reinforcement learning (Mnih et al. 2013) has popularized the use of neural networks as function approximators for learning Q-functions that trained by minimizing the loss function:
[(0) == E (r. + 200 (T++1,
(T arg max Q (T(+1,)) Q5 (T,, ) (Te, To, T(+1) ~ D). (2)
8/"
where i are the parameters of a target network that is copied from 0 periodically to improve stabil-
ity (Mnih et al. 2015) and D is a replay buffer (Lin) 1992) that stores transitions collected by an exploratory policy (lypically e-greedy). Double deep Q-learning (van Hasselt et al. (0166) mitigates overestimation of the learned values by using actions that maximize Q for the target network Qu:
Value Function Factorization Centralized training for decentralized execution (CTDE) has been a major focus in recent efforts in deep multi-agent RL (Lowe et al. [017]] Foerster et al. 2018 Sunc- hag et al. 2018 Rashid al. 2018 Iqbal & Sha 2019). Some work achieves CTDE by introducing methods for factoring Q-functions into monotonic combinations of per-agent utilities, with each w²)).
depending only on single agent's 's history of actions and observations ("(r", """r" This factoriza- tion allows agents to independently maximize their local utility functions in a decentralized manner
with their selected actions combining to form the optimal joint action. This factored representation can only represent a limited subset of all possible value functions (Böhmer et al.] 2020p; however, these methods tend to perform better empirically than those that learn unfactored joint action value functions, most likely because they exploit independence properties among agents (Oliehoek et al.) 2008 Sunehag al. (2018) introduce value decomposition networks (VDN) which decompose the total Valuue as a sum of per-agent utilities: (T, u) E. Q" (r",u"). QMIX (Rashid et al. 2018 extends this approach to use a more expressive factorization. We describe QMIX and how we build our randomized factorization approach on top of it in Section
Attention Mechanisms for MARL Attention models have recently generated intense interest due their ability to incorporate information across large contexts, including the MARL lit- erature (Jiang & Lu 2018 Igbal & Sha 2019 Long et al.] 2020). Importantly for our purposes, they are able to process variable sized sets fixed length vectors (in our case entities). At the core of these models is a parameterized transformation known as multi-head attention (Vaswani et al., 2017). This transformation allows entities to selectively extract information from other entities based on their local context.
We define X as matrix where each row corresponds to an entity (either its state representation or transformed representation of it). The global state can be represented in matrix form as Xe where X = sh Our models consist of entity-wise feedforward layers (denoted as cFF(()) )) and multi-head attention layers (denoted as MHA (A, X, M)). Entity- wise feedforward layers apply an identical linear transformation to all input entities. Multi-heat attention layers serve as a mechanism integrate information across entities. These take in three arguments: the of agents for which to compute output vector A. the matrix X E R (l where d is the dimensionality of the input representations, and mask M E AI The layer outputs a matrix H E IR A| xh where h is the hidden dimension of the layer. The row Ha,, corresponds to a weighted sum of linearly transformed representations from all entities selected by agent Importantly, if the entry of the mask M.,- = 0. then entity e's o's representation cannot be included in H.... Masking serves two important purposes for us: 1) It enables decentralized execution by providing the mask M! agents can only see entities observable by them in the environment, and 2) It enable us to "imagine" the returns among sub-groups of entities. We integrate entity-wise feedforward layers and multi-
yu(s", s such that
3

---

Under review as a conference paper at ICLR 2021
head attention into QMIX in order to adapt it to settings where the number of agents and entities is variable and build our approach from there. The exact process of computing attention layers, as well as the specifies of our attention attention-augmenntr augmented version of QMIX are described in detail in the Appendix.
3 RANDOMIZED ANTITY-WISE FACTORIZATION FOR IMAGINED LEARNING
We now introduce our method, Randomized Entity-wise Factorization for Imagined Learning (REFIL). As discussed in Section 2 value function factorization approaches for coop- erative deep MARL are motivated by their ability to exploit independence between agents while enabling decentralized execution with centralized training. We note that an agent's choice of opti- mal actions is often independent of a subset of its observed entities (cf. soccer breakaway example conjecture that
from Section I in addition the choice of other agents' actions. Furthermore, agents robust to irrelevant entities should be more effective in dynamic environments with variable numbers of agents, as they are better able identify shared patterns of behavior (e.g., breakaways exist in all forms of soccer). We do not know a priori which entities an agent can disregard, so we must consider all possible sub- sub-groups of entities. As such, we propose to factor value functions by imagining returns in random sub- groups.
3.1 METHOD
QMIX (Rashid et al. 2018) relaxes the representational constraints of VDN (Sunehag et al. 2018) by allowing the joint value function Q to be a non-linear monotonic function with respect to the agent-specific utilities Q QTTT = gQQ' (r', ut: (Q)...................)... Q1-11 (714), u| 00): fg). The parameters of the mixing function 0g are generated by hyper-network (Ha et al.) 2017) conditioning on the global state s: Og = h(s; 0n). Every state can therefore have a different mixing function, but the mixing's 's monotonicity maintains decentralizaaiiliyy. as agents can maximize Q without communi- cation. All parameters = { },} are trained with the DQN loss of Equation 2 We extend QMIX with attention layers both to encode variable sized sets of entities observed by each per-agent utility Q" and to mix the utilities of all agents a E A. Partial observability is implemented by a mask My u(s"; s²), Va A, Ve that is provided to attention layers deseribed in section 2
Building on QMIX, for each agent we generate a separate utility that only observes the state features of agents within its randomly selected sub-group: (T (K(Tf,u)), u²), as well as a term that accounts for interactions outside of its group: Q3 (T8: w²), then mixing these 2n (2 for each agent) utilities to form Q"e Importantly, since the mixing network is generated by the full state context, our model can weight factors contextually. For example, if agent a's 's sampled sub-group contains all relevant information to compute its utility such that Q7 Ri Qo, then the mixing network can weight Qi more heavily than Ø8- Otherwise, the network learns to balance Q7 and Q% for each agent, using the full state as context, order to estimate Q We train with these random factorizations in addition to the original QMIX objective. Treating factorization as auxiliary task, rather than as a representational constraint, allows our model to retain the expressivity of QMIX value functions (without sub-group partitions) while exploiting the potential independence between agents and other entities. We note that our auxiliary objective is only used in training, and execution in the environment does not use random factorization.
3.2 IMPLEMENTATION
The mechanism behind our entity-wise factorization relies on simple attention masking procedure. In order to compute in-group utilities Q9 u²) and out-group utilities Q8 v²), we first ran- domly partition all entities E into two disjunct groups (held fixed for an episode), indicated by a random binary vector m E {0, 1J1 The entry m. determines whether entity e is in the first group, and we can take the negation -m, to represent whether e is in the second group. The subset of all agents is denoted as my := imalaeAA From these vectors, we can construct attention masks M E A|x|E]] For example, using the mask M m, mT. T would prevent agents in the first group from "seeing" outside their group since M = 1 only if agent and entity are in the same group. This can be added to a similarly produced mask M -Ma -m. T create M,, a mask that only allows all agents to see the entities within their distinct groups. We construct masks for agents to see within (M,) and out of (Mo) their groups, then combine with observability masks R as such:
M == M A Mg = M A Mo, with M, == mA m T TV-mA-m m Mo = MM (3)
We first draw p E (0,1) uniformly, followed by |8] independent draws from Bernoull(p) distribution.
4

---

Under review as a conference paper at ICLR 2021
("(""" wn)
(19,.)))) u*)
W B
Hypemee
Hyperret
Hypernet
Hlperee
eFF
Mixing Network
e-greedy
1
Softmax
Soffrmax
9 (r",,))
A
MHA
(r3.
MI
eFf
Mo
W1
b
O
Rug
bp
GRU
X
Hypernetwork
FF
Randomized Entity-wise Factorization
MHA
M
m
Wi (M²)
{a}
M
M
imagined
Qu
cFF
M
M
Wi (Mo)
Mo
Agent Utility Network
Figure 2: Schematic for REFIL. Values colored orange or blue are used for computing Q and Q respec- tively. (left) Agent-specific utility networks. These are decentralizable due to the use of observability mask (M*). We include Gated Recurrent Units (Chung et al. (014) retain information across timesteps in order handle partial observability. (top center) Hypernetworks used to generate weights We use a softmax function on the weights across the hidden dimension to enforce non- non-negativit, which we find empirically to be more stable than the standard absolute value function. Hypernetworks are not restricted by observability since they only required during training and not execution. (top right) The mixing network used to calculate Q"d (bottom right) Procedure for performing randomized entity-wise factorization. For masks M, and Mo. colored spaces indicate value (i.e.. the agent designated by the row will be able to see the entity designated by the column), while white spaces indicate a value of O. The color indicates which group the entity belongs to, agents in the red group see red entities in Mr and blue entities in Mo. Agents are split into sub-groups and their utilities calculated for both interactions within their group, well as account for the interactions outside of their group, then monotonically mixed predict Qui
mixing network.
The entry M (a, e] determines both whether agent can see entity and whether entity is in agent a's group; the entry M (a, el is the same but for entities out of a's group. We can use these masks in our attention mechanisms to compute (T u²), which represents the predicted utility of agent a within its group and Q3 (T8, )"), residual term that accounts for the utility of interactions that a would have with the other group.
Given each agent's predicted utility factors for both in-group and out-of-group. we combine these into a Qior such that
the target from the full scenario (y) in (2) using mixing network as in QMIX. This network's 's first layer typically takes n inputs, one for each agent. Since we have 2n factors,
simply concatenate two generated versions of the input layer (using M, and Mo). We then apply the network to the concatenated utilities (7 u²) and Q% u²) of all agents a, compute the predicted value Quet This procedure visualized in Figure and described in more detail in the Appendix.
Our novel approach REFIL uses Q place of Q"a in the DQN loss of (2) to get the auxiliary loss Laux- Our total loss combines both real and auxiliary losses: : CQ + ALmux where a hyper-parameter. In practice, this procedure requires two additional passes through the network (with MD and M as masks instead of M") per training step. These additional passes can be parallelized by computing all necessary quantities in one batch on GPU. is feasible to split entities into an arbitrary number i of random sub-groups without using more computation by sampling several disjunct vectors m and combining them them in the same way as we combine m and -m in Equation 3 to form M, and Mo. Doing could potentially bias agents towards considering smaller subsets of entities.
EXPERIMENTAL RESULTS
In our experiments, we aim to justify the main components of REFIL: 1) randomized sub-group factorization and 2) training as auxiliary objective. We begin with experiments in simple domain we construct such that agents' decisions rely only on a subset of all entities, and that subset is known, compare our approach to approaches that use this domain knowledge. Then, move
5

---

Under review as a conference paper at ICLR 2021
1.-
Random Initialization
Goal
644
Narr
RETIL
/Food
REFIL (Rasdoriedl Ovids)
60-
010
0.75
(a) Game Visualization
(b) Win Rate over Time
Figure 3: Group Matching Game. We use the values na 8, ne 6, and ng 2 in experiments. Shaded
region 95% confidence interval across runs.
on to testing on complex StarCraft micromanagement tasks to demonstrate our method's ability to scale to complex domains.
4.1 GROUP MATCHING GAME
We construct group matching game, pictured in Figure 3a where each agent only needs consider a subset of other agents to act effectively and we know that subset as ground-truth (unlike more complex domains such StarCraft). As such, the task can be described as follows: Agents (of which there are n.,) are randomly placed in one of n., cells and assigned to one of ng groups (represented by the different colors) the start of each episode. They can choose from three actions: move clockwise, stay, and move counter-clockwise. Their ultimate goal is to be located in the same cell as the rest of their group members, at which point an episode ends. There is no restriction on which cell agents form a group in (e.g., both groups can form in the same cell). All agents share a reward of 2.5 when any group is completed (and an equivalent penalty for formed group breaking) as well as a penalty of -0.1 for each time step in order to encourage agents to solve the task as quickly as possible. Agents' entity-state descriptions
include the cell that the agent currently occupying as well as the group it belongs to (both one-hot encoded), and the task is fully-observable. Notably, agents can act optimally while ignoring agents outside of their group.
Ground-truth knowledge of relevant entities enables us disentangle two aspects of our approach: the use of entity-wise factorization in general and specifically using randomly selected factors. We construct two approaches that use this knowledge to build factoring masks M, and Mo which are used in place of randomly sampled groups (otherwise the methods are identical to REFIL). REFIL (Fixed Oracle) directly uses the ground truth group assignments (different at each episode) to build masks. REFIL (Randomized Oracle) randomly samples sub-groups from the ground truth groups only, rather than from all possible entities. We additionally train REFIL and QMIX (Atlention) (i.e., REFIL with no auxiliary loss).
Figure 3b shows that using domain knowledge alone does not significantly improve performance in this domain (QMIX (Attention) vs. REFIL (Fixed Oracle)). In fact our randomized factorization approach is able to surpass the use domain knowledge. The randomization therefore to be crucial. One hypothesis for this phenomenon is that randomization of sub- sub-group factors enables better knowledge sharing across diverse settings (in this case unique group assign- ments). For example, the situation of two agents from the same group being located in adjacent cells occurs within all possible group assignments. If sampling randomly, our approach will occasionally sample these two agents alone in their own group. Even if the rest of the context in given episode has never been seen by the model before, as long as this sub-scenario has been seen, the model has some indication of the value associated with each action. Even when restricting the set of entities to form sub-groups with to those that we know can be relevant to each agent (REFIL (Randomized Oracle)) find that performance does significantly improve. These results suggest that ran- domized sub-group formation for REFIL is viable strategy (vs attempting to learn which entities relevant and selecting sub- sub-groups from there), and the main benefit of our approach is promote generalization across scenarios by breaking value function predictions into reusable components.
REFIL appears

---

Under review as a conference paper at ICLR 2021
4.2 STARCRAFT
We next test on the StarCraft multi-agent challenge (SMAC) (Samvelyan et al. 2019). The tasks in SMAC involve micromanagement of units in order to defeat a set of enemy units in battle. Specif- ically, we extend SMAC to settings with variable types and quantities of agents. We hypothesize that our approach is especially beneficial in this setting, it should encourage of models to iden- tify independence between entities and generalize to more diverse settings as a result. The dynamic setting requires some small modifications to SMAC, though little as possible to maintain the challenging nature of the tasks. In the standard version of SMAC, both state and action spaces depend fixed number of agents and enemies, so our modifications, discussed in detail in the appendix, alleviate these problems.
aim to change the environment as
In our tests we evaluate on three settings we call 3-8sz, 3-8esz, and 3- 3-8MMM. 3-8sz pits symmetrical teams of between 3 and 8 agents against each other where the agents are a combination of Zealots and Stalkers (similar to the 2s3z and 3s5z tasks in the original SMAC). 3-8csz pits symmetrical teams of between O and 2 Colossi and 3 to 6 Stalkers/Zealots against each other (similar to Ic3s5z). 3-8MMM pits symmetrical teams of between 0 and 2 Medics and 3 to 6 Marines/Marauders against each other (similar to MMM and MMM2). As sanity check, we additionally modify our approach to work with non-attention models such that we can test on the original SMAC tasks against existing methods. These results (located in the appendix) show that we can significantly improve on QMIX (previously state-of-the-art) in 2 of 3 settings tested.
Ablations and Baselines We introduce several ablations of our method, well as adaptations of existing methods to handle variable sized inputs. These comparisons are summarized in Table [I] QMIX (Atten- tion) is our method without the auxiliary loss. REFIL (VDN) is our approach using summation to combine all factors (a la Value Decomposition Networks (Sunehag et al.) 2018) rather than a non-linear monotonic mixing network. VDN (Attention) does not include the auxiliary loss and uses summa- tion factor mixing. QMIX (Mean Pooling) is QMIX (Attention) with attention layers re- placed by mean pooling. We also test max
Table 1: Comparison of tested methods.
Imagined Leamnngg
Eatity Aggregation Method
Base
Name
Algorithm
REFIL
MHA
QMIX
QMIX (Allention)
MHA
QMIX
REFIL (VDN)
MILA
VDN
VDN (Attention)
MHA
VDN
QMIX (Max Pooling)
Max Pool
QMIX
QMIX (EMP)
EMP4
QMIX
ROMA (Attention)
MHA
ROMA 5
Qatteu (Attention)
MHA
Qatien 9 QTRAN
QTRAN (Attention)
MHA
Vaswani al. 2017
Rashid et Wang el
2018 Sunchae al (0118) Yang et al. (2020)
Acarwal a
2020
Son et (209
pooling but find the performance to be marginally worse than mean pooling. Importantly, for pool- ing layers we add entity-wise linear transformations prior to the pooling operations such that the total number of parameters is comparable to attention layers.
For baselines we consider some follow-up works to QMIX that attempt to improve the mixing net- work to be more expressive: QTRAN (Son et al., (019)) and Qatten (Yang et al. (020). We addition- ally consider an alternative mechanism for aggregating information across variable sets of entities, known as Entity Message Passing (EMP) (Agarwal et al. 2019). We specifically use the restricted communication setting where agents can only communicate with agents they observe, and we set the number of message passing steps to 3. Finally, we compare to a method that builds QMIX by attempting to learn dynamic roles that depend on the context each agent observes: ROMA (Wang) et al. 2020a). For all approaches designed for the standard SMAC setting, we extend them with the same huad attention architecture that our approach uses.
Results and Discussion Our results on challenges in dynamic STARCRAFT settings can be found in Figure 4
We find that REFIL outperforms all ablations consistently in these settings. REFIL (VDN) performs much worse than our approach and VDN (Attention), highlighting the importance of the mixing network to handle contextual dependencies between entity partitions. Since the trajectory of a subset of entities can play out differently based on the surrounding context, it's important for our factorization approach to recognize and adjust for these situations. The mixing network handles these dependencies by a) incorporating global state information into the mixing procedure, and b) mixing utilities in a non-linear monotonic fashion, rather than summing as in VDN. As such, the increased representative capacity of the QMIX mixing network, relative to VDN, is crucial. The use of mean-pooling in place of attention also performs poorly, indicating that attention is valuable for aggregating information from variable length sets of entities.
7

---

Under review as a conference paper at ICLR 2021
REFIL
05-
QUX (Azeida)
(VON)
VEN
100% 1oas
QUX (Yoca Puslng
112
012
001
02
05
Steps
(a) 3-8sz
(b) 3-8csz
(c) 3-8MMM
REFH
QYDC (2VP)
KIMA (Atsmter)
Uetten (Irretto)
OTRAN (Acartio))
Uea
+10
(d) 3-8s7
(c) 3-8esz
(f) 3-8MMM
Figure 4: Test win rate over time variable entity STARCRAFT environments. Shaded region is 95% confidence interval across 6 runs. (top row) Ablations of method. (bottom row) Baseline methods.
With respect to the baselines, also find that REFIL consistently outperforms other methods, highlighting the unique challenge of learning in such dynamic settings where entity types are variable at each episode. The improvements that ROMA, Qatten, and QTRAN seen in other settings over QMIX, do not appear to manifest themselves in this setting. Moreover, the entity aggrega- tion method of EMP does not improve performance over the standard MHA module that we use, likely due to the fact that EMP is most effective in settings where partial observability is major hindrance to successful task com- pletion. In this way, the target of EMP and REFIL are opposite, as the goal of REFIL is to ignore extrancous information when possible during training to improve knowledge transfer.
Nane
0.0
0.8
Win Rats
0.6
0.2
Skeps
Figure 5: Varying for REFIL 3-8sz.
In order to understand the role of training as an auxiliary objective (rather than entirely replacing the objective) vary the value of to interpolate between two modes: À = 0 is simply QMIX (Attention), while X trains exclusively with random factorization. Our results (Figure show that, similar regularization methods such Dropout (Srivastava al. (2014), there is sweet spot where performance maximized before collapsing catastrophically. Training exclusively with random factorization does learn anything significant. This fallure likely due the fact that we use the full context in our targets for learning with imagined scenarios as well as when executing our policies, we still need to learn with it in training.
Finally, we consider a qualitative experiment to highlight the sort of common patterns that REFIL able leverage (Figure Zealots (the only melee unit present) are weak to Colossi, so they learn to hang back and let other units engage first. Then, they jump in and intercept the enemy Zealots while all other enemy units are preoccupied, leading to a common pattern skirmish (highlighted at t=15). REFII. enables behaviors learned in these types of sub-groups to be applied more effectively across all unique unit type configurations. By sampling groups from all entities randomly, we will occasionally end up with sub-groups that include only Zealots, and the value function predictions learned in these sub-groups can be applied not only to the episode at hand, but any episode where similar pattern emerges.
Zealot vs. Zealot
5 RELATED WORK
Multi ageti- reinforcement learning (MARL) is a broad field encompassing cooperative (Foerster
al. 2018 Rashid al. 2018: Sunnhaa al. 2018), competitive (Bansal et al. 2018 Lanctot
8

---

Under review as a conference paper at ICLR 2021
Key
Agent
Enemy
Ep 1
Stalber
Colosses
Ep 2
Unit Stiedd
Usit Hhatth
Indicatical an sgent's stteck Length commaponds time undil attack
Ep
Figure 6: Simplified rendering of common pattern that emerges across initializations in the 3-8csz SMAC setting, highlighted t 15. REFIL enables learning from each episode to inform behavior in the others.
ct al. 2017), and mixed (Lowe et al., 2017: Iqbal & Sha 2019) settings. This paper focuses on cooperative MARL with centralized training and decentralized execution (Oliehoek et al. 2016 CTDE). Our approach utilizes value function factorization, an approach aiming to simultaneously overcome limitations of both joint Hausknecht (2016) and independent learning CCaus& & Boutilier (1998) paradigms. Early attempts at value function factorisation require apriori knowledge of suit- able per-agent team reward decompositions over local compositions of individual Q-value functions learnt from individual reward functions (Schneider al., [199)), as well summing individual Q-functions with individual rewards before greedy joint action selection (Russell & Zimdars 2003). Guestrin et al. (2002); Kok & Vlassis
interaction dependencies. These include optimising
(2006) factorise the total Q-value function using coordination graphs based on interaction depen- dencies for the task at hand, similarly to max-plus approaches Kuyer et al. (2008;; Pol & Olichoek 2016). Recent approaches from cooperative deep multi-agent RL allow for value factorisations to be learnt from experience from a single team reward function and no prior knowledge of interaction dependencies. Valoe-Decoopoostioon Networks (VDN) (Sunehag et al.] (2088) decompose the joint of local utility functions used for greedy action selection. QMIX Rashid ctt al. (2018) extends such additive decompositions to general monotonic functions. Several works
Q-value function into
extend QMIX improve the expressivity of mixing functions (Son et al. [019 Yang al.. 2020),
learn latent embeddings to help exploration (Mahajan et al. 2019) or learn dynamic roles (Wang
al. 2020a), and encode knowledge of action semantics into network architectures (Wang al.
2020b
Several recent works have addressed the topic of generalization and transfer across environments with varying agent quantities, though the learning paradigms considered and assumptions made differ from our approach. Carion et al.] (2019) devise an approach for assigning agents to tasks, assuming the existence of low-level controllers to carry out the tasks, and show that it can scale to much larger scenarios than those seen in training. Burden (2020) propose a transfer learning approach using convolutional neural networks and grid-based state representations to scale to sce- narios of arbitrary size. |Agarwal et al. (2019) introduce an entity message passing framework to enable agents to attend to specific entities, of which there may be variable amount, based on their local context, similar to the multi-hear attention module we use in our approach. Several approaches devise attention or graph-neural
based models for handling variable sized inputs and focus on learning curricula to progress on increasingly large/challenging settings (Long et al. 20200 Baker al. 2019 Wang al. 2020c). In contrast to these curriculum learning approaches, training simultaneously on scenarios of varying sizes and specifically focus on developing a training paradigm for improving knowledge sharing across such settings accelerate learning.
focus
CONCLUSION
In this paper we consider MARL setting where we aim to learn policies to control teams of agents in scenarios with varying types and quantities entities. We propose REFIL, an approach that reg- ularizes value functions to identify independence relationships between entities, in turn promoting generalization and knowledge transfer within and across multi-agent settings with varying quanti- ties of agents. Our results show that our contributions yield performance improvements in complex cooperative tasks. In future work, we hope to explore alternative methods for learning independence relationships between entities beyond randomized partitions.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprint arXiv:1906.01202 2019.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2019.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. In International Conference Learning Representations, 2018.
Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Proceedings of Machine Learning and Systems (ICML), pp. 2611-2622, 2020. URL orq/ abs/ exq/abs//e
1910. abs//asl.
00091
Nicholas Burden. Deep Multi-Agent Reinforcement Learning in Starcraft II. Master's thesis, Uni- versity Oxford, 2020.
Nicolas Carion, Nicolas Usunier, Gabriel Synnaeve, and Alessandro Lazaric. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8 8128-8138, 2019.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning. December 2014, 2014.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/AAА,, 1998(746-752);2 1998.
Djnek-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 15 2015.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, 2018.
Carlos Guestrin, Michail G. Lagoudakis, and Ronald Parr. Coordinated Reinforcement Learning. In Proceedings of the Nineteenth International Conference on Machine Learming, ICML '02, pp. 227-234, San Francisco, CA, USA, July 2002. Morgan Kaufmann Publishers Inc. ISBN 978-1- 55860-873-3.
David Ha, Andrew M Dai, and Quoc V Le. Hypernetworks. In International Conference on Learn- ing Representations, 2017.
Matthew Hausknecht and Peter Stone. Deep recurrent g-leariing for partially observable mdps. In 2015 AAAI Fall Symposia, pp. 29-37, 2015. php/FSs5
happ/FSS/ET Fwt.S//S/SSSSeeeeeieeees
Matthew John Hausknecht. Cooperation and Communication in Multiagent Deep Reinforcement Learning. PhD thesis, The University of Texas at Austin, 2016.
Shariq Iqbal and Fei Sha. Actor-aatentiooncciitic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2961-2970, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL httpttt::pppcceeeiiiiii html
mlr.
pprss/v97/igball
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation.
arXiv preprint arXiv: /003.0775373,.. 2018.
Jelle R. Kok and Nikos Vlassis. Collaborative Multiagent Reinforcement Learning by Payoff Prop- agation. The Journal of Machine Learning Research, 7:1789-18 -12881 December 2006. ISSN 1532- 4435.
10

---

Under review as a conference paper at ICLR 2021
Daphne Koller and Ronald Parr. Computing factored value functions for policies in structured mdps. In IJCAI, volume 99, Pp. 1332-13383 1999.
Lior Kuyer, Shimon Whiteson, Bram Bakker, and Nikos Vlassis. Multiagent reinforcement learning for urban traffic control using coordination graphs. Proceedings of the 2008th European Con- ference Machine Learning and Knowledge Discovery in Databases Volume Part I, ECMLP- KDD 08. pp. 656-671, Berlin, Heidelberg, September 2008. Springer-Verlag ISBN 978-3-540- 87478-2.
Mare Lanetot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Per- olat, David Silver, and Thore Graepel. A unified Kame-Theoretee approach to multiagent rein- Guyon, U V Luxburg, Bengio, Wallach, Fergus, Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4193-4206. Curran Associates, Inc., 2017.
forcement learning.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention-base permutation-ing invarrinn neural networks. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753, Long Beach, California, USA, 09-15 Jun 2019, PMLR. URL http:::pppcceeee/ee/
mlrr.
Long-Ji Lin. Self-improving reactive agents based on reinforcement leaming, planning and teaching. Machine Learning, 83))233--31, 1992.
Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary population curriculum for scaling multi-agent reinforcement leaming. In International Confer- ence on Learning Representations, 2020. URL
new.//ne/////e wwtp:e/wwweeeeeeteeeeeee forummidd
SJxbHkrKDH
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiive environments. Advances in Neural Information Processing Systems, pp. 6382-6382 0222,6333,222 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7613-7624,
2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Frans A Oliehoek, Matthijs TJ Spaan, Nikos Vlassis, and Shimon Whiteson. Exploiting locality of interaction factored dec-pomdps. In Int. Joint Conf. on Autonomous Agents and Multi-Agent Systems, pp. 517-524, 2008.
Frans A Oliehoek, Christopher Amato, al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Elise van der Pol and Frans A. Oliehock. Coordinated Deep Reinforcement Learners for Traffic Light Control, 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent rein- forcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4295-4304, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018.
Stuart Russell and Andrew L. Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the Twentieth International Conference on International Conference Machine Learning. ICML '03, pp. 656-663, Washington, DC, USA, August 2003. AAAI Press. ISBN 978-1-57735-188-4
11

---

Under review as a conference paper at ICLR 2021
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr. Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp- 2186-2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Jeff Schneider, Weng-Keen Wong, Andrew Moore, and Martin Riedmiller. Distributed Value Func- tions. In In Proceedings of the Sixteenth International Conference on Machine Learning, Pp- 371-378. Morgan Kaufmann, 1999.
Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Bochmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement
learning. In Advances Neural Information Processing Systems 32. pp. 9927-
9939. Curran Associates, Inc., 2019. URL http: capppprrs..pppp. nips https.ccc/aaee capparer/////////// common-t-oooo-oooo know
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 5887-5896, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1229-1958 2014.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Proceed- ings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AA- MAS 18, pp. 2085-2087, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the 13th AAAI Conference on Artificial Intelligence, pp. 2094-2100, 2016. /RRp:ttps://aat
org 250.906469.......
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 6000-6010, 2017.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference Machine Learning, 2020a.
Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. In International Conference on Learning Representations, 2020b. URL httpp:://penrrvview net fot.:ettoooorr//drrrrrgg
Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In AAAI Conference on Artificial Intelligence, 2020c.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.0399: 2020.
12

---

Under review as a conference paper at ICLR 2021
A ATTENTION LAYERS AND MODELS
Attention models have recently generated intense interest due to their ability incorporate informa- tion across large contexts. Importantly for our purposes, they are able to process variable sized sets of inputs.
We now formally define the building blocks of our attention models. Given the input X, matrix where the rows correspond to entities, we define an entity-wise feedforward layer as standard fully connected layer that operates independently and identically over entities:
( W.b. b) XW b E W E Rddh E R
(4)
Now, we specify the operation that defines attention head, given the additional inputs of S S 2(,="", a set of indices that selects which rows of the input X are used to compute queries such that Xs E RlS)xdd and M, a binary obserability mask specifying which entities each query entity can observe (i.e. M.j = 1 when i E S can incorporate information from j E Z[l.a"] into its local context):
QKI Atten(S, M; W W WV) softmax maskk
.MMM) V RISixA
('h Vh= vh
(5)
Q Xs, K- XW = M E (0, 1jisi"", w W W E gaxh
(6)
The mask(Y.. .M) operation takes two equal sized matrices and fills the entries of Y with -00 in the indices where M is equal to 0. After the softmax, these entries become zero, thus preventing the attention mechanism from attending to specific entities. This masking procedure is used in our case to uphold partial observability. Only one attention layer is permitted in the decentralized execution setting; otherwise information from unseen agents can be propagated through agents that are seen. W WK and W are all learnable parameters of this layer. Queries, Q. can be thought of as vectors specifying the type of information that an entity would like select from others, while keys, K, can be thought of as specifying the type of information that an entity possesses, and finally, values, V, hold the information that is actually shared with other entities.
We define multi-head heii-head--ttee attention as the parallel computation of attention heads as such:
MHA (S, x, M) = concat (Atten S, s, W w.K Wy) .j (1 (1...."))
(7)
The size of the parameters of an attention layer does not depend on the number of input entities. Furthermore, we receive an output vector for each query vector.
B AUGMENTING QMIX WITH ATTENTION
The standard QMIX algorithm relies on a fixed number of entities in three places: inputs of the agent-specific utility functions Qa. inputs of the hypernetoork, and the number of utilities entering the mixing network, that is, the output of the hypernetorr. QMIX uses multi-layer perceptrons for which all these quantities have to be of fixed size. In order to adapt QMIX the variable agent quantity setting, such that we can apply single model across all episodes, we require components that accept variable sized sets of entities as inputs. By utilizing attention mechanisms, we can design components that are no longer dependent
fixed number of entities taken as input. We define the following inputs: X := sf 1 I i < d, e E E; M := uus""ss²) E A, e E E. The matrix X is the global state reshaped into matrix with a row for each entity, and M is binary observability matrix which enables decentralized exccution, determining which entities are visible to each agent.
B.1 UTILITY NETWORKS
While the standard agent utility functions map a flat observation, whose size depends on the number of entities in the environment,
utility for each action, our attention-utille functions take in a variable sized set of entities and return a utility for each action. The attention layer output for agent is computed as MHA (fa), X, MP), where X is row-wise transformation of X (e.g..
13

---

Under review as a conference paper at ICLR 2021
an entity-wise feedforward layer). If agents share parameters, the layer can be computed in parallel for all agents by providing A instead of {a), which we do in practice.
B.2 GENERATING DYNAMIC SIZED MIXING NETWORKS
Another challenge in devising QMIX algorithm for variable agent quantities adapt the hyper- networks that generate weights for the mixing network. Since the mixing network takes in utilities from each agent, we must generate feedforward mixing network parameters that change in size depending the number of agents present, while incorporating global state information. Conve- niently, the number of output vectors of MHA layer depends on the cardinality of input set S and can therefore generate mixing parameters of the correct size by using S A and concatenating the vectors to form a matrix with one dimension size depending on the number of agents and the other depending the number of hidden dimensions. Attention-based QMIX (QMIX (Attention)) trains these models using the standard DQN loss in Equation
Our two layer mixing network requires the following parameters to be generated: W1 R+(1AIxA"") by E RAa W2 E TR*+(""),, b2 E IR, where hm is the hidden dimension of the mixing network and |A| is the set of agents.
Note from Eq. (5) that the output size of the layer is dependent on the size of the query set. As such, using attention layers, we can generate a matrix of size |A] x hTm by specifying the set of agents, A, as the set of queries S from Eq. (5). We do not need observability masking since hypernetworks are only used during training and can be fully centralized. For each of the four components of the mixing network (W1, b W2, b2), we introduce a hypernetwork that generates parameters of the correct size. Thus, for the parameters that are vectors (b, and W2), we average the matrix generated by the attention layer across the |A| sized dimension, and for b2, we average all elements. This procedure enables the dynamic generation of mixing networks whose input size varies with the number of agents, Assuming q - [Q w²), ..... (T" u")], then Q is computed as:
Q'a u) = o((q W1) + 6 W22 + b2
(8)
where a is an ELU nonlinearity (Clevert et al. 2015).
C ENVIRONMENT DETAILS
C.I STARCRAFT WITH VARIABLE AGENTS AND ENEMIES
The standard version of SMAC loads map files with pre-defined and fixed unit types, where the global state and observations are flat vectors with segments corresponding to each agent and enemy. Partial observability is implemented by zeroing out segments of the observations corresponding to unobserved agents. The size of these vectors changes depending on the number of agents placed in the map file. Furthermore, the action space consists of movement actions as well as separate actions to attack each enemy unit. As such the action space also changes as the number of agents changes.
Our version loads empty map files and programmatically generates agents, allowing greater flex- ibility in terms of the units present to begin each episode. The global state is split into a list of equal-sized entity descriptor vectors (for both agents and enemies), and partial observability is han- dled by generating matrix that shows what entities are visible to each agent. The variable-ste size action space handled by randomly assigning each enemy tag at the beginning and designating action to attack each possible tag, of which there are a maximum number (i.e. the maximum possible number of enemies across all initializations). Agents are able to see the tag of the enemies they observe and can select the appropriate action that matches this tag in order to attack a specific enemy.
each episode
D EXPERIMENTAL DETAILS
Our experiments were performed on a desktop machine with 6-core Intel Core i7-6800K CPU and 3 NVIDIA Titan Xp GPUs, and a server with 2 16-core Intel Xeon Gold 6154 CPUs and 10 NVIDIA Titan Xp GPUs. Each experiment is run with 8 parallel environments for data collection and a single GPU. REFIL takes about 24 hours run for 10M steps on STARCRAFT. QMIX (Attention) takes
14

---

Under review as a conference paper at ICLR 2021
about 16 hours for the same number of steps STARCRAFT. Reported times the desktop machine and the server runs approximately 15% faster due to more cores being available for running the environments in parallel.
E HYPERPARAMETERS
Hyperparameters were based on the PyMARL (Samvelyan et al. 2019) implementation of QMIX and are listed in Table? 2 All hyperparameters are the same all STARCRAFT settings. Since we train for 10 million timesteps (as opposed to the typical 2 million in standard SMAC), we extend the epsilon annealing period (for spsilon-greedy exploration) from 50,000 steps 500,000 steps. For to our approach (hidden dimensions of attention layers, number of attention heads, weighting of imagined loss), the specified values in Table 2 were the first values tried, and found them to work well. The robustmess of our approach to hyperparameter settings, well as the fact that we do not tune hyperparameters per environment, is a strong indicator of the general applicability of our method.
hyperparameters
Table 2: Hyperparameter settings across all runs and algorihhs/bassllnes.
Name
Description
Value
Ir optimizer optim
learning rate
0.0005
type of optimizer RMSProp param
RMSProp
0.99
optim e
RMSProp param
le -
target update interval bs
copy live params to target params every episodes
200
batch size (# of episodes per batch)
32
grad clip
reduce global norm of gradients beyond this value
10
[D]
maximum size of replay buffer (in episodes) discount factor
5000
Y
0.99
starting e
starting value for exploraton rate annealing ending value for exploraton rate annealing
1.0
ending
0.05
anneal time
number of steps to anneal exploration rate over
500000
hidden dimensions for attention layers hidden dimensions for RNN layers
128
h"
64
hidden dimensions for mixing network Number of attention heads type of nonlinearity (outside of mixing net)
32
attention heads
nonlinearity
ReLU
A
Weighting between standard QMIX loss and imagined loss
0.5
Tieleman & Hinton (2012)
F ADDITIONAL RESULTS
We test a modified non-attention version of our approach along with state of the art methods on the standard version of SMAC, where entity types are constant at the start of each episode. Since the number and type of agents and enemies is constant at each episode, observations and states can represented as fixed-size vectors. We can thus use MLPs as models (as is standard in the literature) for these tasks and adapt our approach to suit this setting while comparing to unmodified versions of existing approaches. Rather than masking an attention mechanism, we simply zero out the features in the observations and states that correspond to entities would like to mask out. These experiments are performed in order to compare our approach to results validated in the literature.
We compare against QMIX (Rashid et al. 2018) and VDN (Sunehag et al. 2018). as well as an ablation of our approach that uses additive mixing (a la VDN) of entity partition factors instead of mixing network which we call REFIL (VDN). We use the architectures and hyperparameters from the QMIX (Rashid al. 2018 paper in these settings.
15

---

Under review as a conference paper at ICLR 2021
01-
Tome
02-
REAL (OM)
112
Vie
64-
5500 one Takio veleurtion memorr/ooooooooooooooooo
1900 onioe Tokine womeeoeeeeeeeeeeeo
05
Steps
Stp
Steps
(a) 3s5z
(b) 2c vs 64zg
(c) 3s5z vs 3s6z
Figure Results on SMAC.
Results can be found in Figure 7 While we expect REFIL to be most effective in the setting of varying types and quantities of agents, we still find that it improves on QMIX in 2 of the 3 scenarios tested. In the standard SMAC benchmarks, we find our approach is able to match or outperform the best baseline across all settings. Specifically, our factorization method (which builds on QMIX) improves the performance of QMIX in of settings tested. As far as we are aware REFIL outperforms all reported results in the literature on "2c vs 64 zg" (clasified SMAC). The relative improvement over QMIX, combined with the fact that it does not ever appear to hurt performance, indicates that the benefits of our method are not limited to settings with varying types and quantities of agents, though the positive effects are more pronounced there.
"hard" task in
16


Reviews:
[{"review_id": "oW-6pS6IGRq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a randomized entity-based attentional mechanism to regularize the observation space for efficient multi-agent reinforcement learning. Specifically, the authors expect that their method can help agents focus on entities that are relevant to their decision-making process. The aim of the paper is well-positioned in multi-agent settings and are expected to help improve performance by exploiting the loosely coupled structure of multi-agent tasks (although the authors do not explicitly model the decision dependency among agents). However, I have some doubts about whether the proposed method can address the target of the paper.\n\nIntuitively, in multi-agent settings, the number of entities is at least the number of agents, say O(n). O(n) is a quite optimistic estimation because we even do not consider other entities. If the authors want to find an optimal bi-partition over the entity space for each agent, the search space is at least O(2^n), which grows exponentially with the number of agents. To design an efficient search algorithm over such a large space needs to take advantage of some well-designed inductive bias or heuristics. The authors use a random strategy here, which, in my opinion, is not sufficient to guarantee a satisfactory solution. There is no denying that randomization can give a good solution in some cases, but this can not be held as a general rule. Even if the bi-partition structure is trained end-to-end, I also suspect that learning such a structure is not easier than learning from scratch.\n\nAdditionally, I think the authors largely ignore the contribution of a related work [Agarwal et al., 2020]. Although they cite this paper in Sec. 5, unlike what is stated in this paper, the main contribution of [Agarwal et al., 2020] is a GNN-based attentional mechanism over entity spaces. They propose to let agents learn to attend to different entities under different observations. In this way, the target of [Agarwal et al., 2020] and this paper largely overlap. I was expecting that the authors provide a thorough comparison with [Agarwal et al., 2020] in their experiments. If the authors can demonstrate that their method can outperform [Agarwal et al., 2020], I will consider improve my rating.\n\n\n[Agarwal et al., 2020] Agarwal, A., Kumar, S., Sycara, K. and Lewis, M., 2020, May. Learning Transferable Cooperative Behavior in Multi-Agent Teams. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (pp. 1741-1743).", "extra_scores": {}, "extra_text": {}}, {"review_id": "RgP1vHeKUDw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for randomized factorization of multi-agents for efficient learning. The idea is inspired by the presence of irrelevant entities present in an agent's observational view and how removing those could aid the learning process.\nAgents are randomly divided into groups so that an agent can separately measure the influence of entities present in the same group and the entities present in the other groups. Since the groups are randomized, this helps the agent to create groups of variable size based on its utility prediction of in-group and out-of-group entities.\n\nAlthough the paper only uses two groups for derivation and experiments, it claims that the same method can be applied to more than two groups but is yet to be demonstrated.\n\n\nThe paper is easy to read and the figures are excellent and self-explanatory. The environments chosen are also indicative of the importance of each component. The authors also found that randomized factoring performs better empirically than using domain knowledge while the state-of-the-art lies in the combination of the two. Further, the idea of training variable-sized hypernetwork is quite fascinating and fits well in the overall framework.\n\nSome questions:\nFigure 3b does not show results till convergence, please put the entire plots.\n\nI think it would be better if the SMAC setup is explained in more detail. I couldn't understand how the tagging is done (in Appendix). Does it deterministically tie an action to an enemy?\n\nQTRAN has been shown to perform better than QMIX in competitive domains. I would encourage the authors to compare the results with this baseline too.", "extra_scores": {}, "extra_text": {}}, {"review_id": "81o2hcEfXDE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an observation factorization method to avoid the influence of the irrelevant part on value estimation. Specifically, they design an entity-wise attention network with a masking procedure. This network is used to filter the irrelevant part of the original observation of each agent. Then the output is used to estimate the individual q-value, as well as input to the mixing network to generate the Q_tot. Two kinds of Q_tot are trained together by combing two loss functions linearly with a hyper-parameter. Experimental results show REFIL combined with QMIX surpasses vanilla QMIX and VDN in several SMAC scenarios.\n\nThis paper is related to the topics of ICLR. However, I think the related work is not sufficient to cover the background. More detail comments can be found below. \n\n*****Some specific comments:*****\n\nIt is not clear that what is the initialization of two masks, and how to update the masks. \n\nThe authors mentioned there are two groups of entities. However, the entity type is also unclear. I guess SMAC only contains two entity types: alive agents and died agents? How to represent an entity inactive?\n\nOne question is why just consider two kinds of groups, what would happen if there exist more than two groups for all entities. In SMAC or soccer, it does contain more than two common patterns. Furthermore, it seems that the masking procedure is hard to extend to the situation with a larger number of groups.\n\nActually, I think REFIL is similar to ROMA [1] and ASN [2] in different ways. First, REFIL considers two kinds of groups corresponding to a simple version of ROMA which has two roles. Second, REFIL does the same thing as ASN that learns the value estimation by considering a more useful part of the observation. ASN directly divide the observation based on the action semantics, while REFIL tries to learn a suitable observation factorization through entity-wise attention with masking. However, these two very relevant works are not discussed and compared in this paper.\n\nSome suggestions,\n\nI think current experiments could not well support motivation. If authors show some examples in SMAC that what kinds of common patterns agents learn would be better to support this idea.\n\nSince REFIL can be integrated into current MARL algorithms, it is better to consider more recent published MARL methods as baselines, such as QTRAN, QATTEN, QPLEX.\n\n[1] Roma: Multi-agent reinforcement learning with emergent roles. ICML. 2020.\n\n[2] Action Semantics Network: Considering the Effects of Actions in Multiagent Systems. ICLR. 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7znBztGeoJg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to incorporate a masked attention mechanism in QMIX for value function factorization to disentangle value predictions from irrelevant agents/entities. The masking is based on a random sampling from the whole set of agents to from random subsets, based on which it can compute within-group and without-group Q-functions. The method is able to handle varying types and number of agents. The paper conducts experiments on a simple game to understand the effect, and then test on 3 SMAC games, which shows the effectiveness of the proposed REFIL method.\n\nStrong points:\n- The paper is well-written and clear.\n- The paper studies an important topic in MARL, i.e., how to deal with varying types and number of agents, and propose a simple yet effective approach, which incorporates attention mechanism in QMIX with random masking.\n- Experiments on several games illustrate the effectiveness of the method, with proper ablation study to understand the importance of each component (attention, mixing network, random masking).\n\nConcerns:\nThe main focus of the paper is to “disentangle value predictions from irrelevant entities”. However, not only does REFIL relies on Q_I (within-group Q-function), but also it relies on Q_O (without group Q-function). If the agent successfully learn this neglect of irrelevant entities, focusing on Q_I would be enough, without triggering additional computation of unnecessary Q_O. Could authors better explain this? In addition, consider the breakaway example, as the attacker has only to focus on the goal keeper, is the random sampling scheme from all agents effective compared with counterparts that only need to focus on the goalkeeper? Could authors conduct additional experiments on football to better support the claim?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an attention based technique to focus on relevant entities in multi-agent reinforcement learning.  While the effectiveness of the proposed method is demonstrated on some tasks, there remain major concerns including the following:
1. It is not sufficiently convincing that the proposed method performs well in more complex domains
2. Novelty over Agarwal et al. and MAAC is rather minor
Decision: Reject