UPDeT: Universal Multi-agent RL via Policy Decoupling with Transformers
Authors: Siyi Hu
Fengda Zhu
Xiaojun Chang
Xiaodan Liang
Venue: ICLR 2021 Spotlight (2021)
Keywords: Multi-agent Reinforcement Learning
Transfer Learning

Abstract: Recent advances in multi-agent reinforcement learning have been largely limited in training one model from scratch for every new task. The limitation is due to the restricted model architecture related to fixed input and output dimensions. This hinders the experience accumulation and transfer of the learned agent over tasks with diverse levels of difficulty (e.g. 3 vs 3 or 5 vs 6 multi-agent games).  In this paper, we make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing one single architecture to fit tasks with the requirement of different observation and action configurations. Unlike previous RNN-based models, we utilize a transformer-based model to generate a flexible policy by decoupling the policy distribution from the intertwined input observation with an importance weight measured by the merits of the self-attention mechanism. Compared to a standard transformer block, the proposed model, named as Universal Policy Decoupling Transformer (UPDeT), further relaxes the action restriction and makes the multi-agent task's decision process more explainable. UPDeT is general enough to be plugged into any multi-agent reinforcement learning pipeline and equip them with strong generalization abilities that enables the handling of multiple tasks at a time. Extensive experiments on large-scale SMAC multi-agent competitive games demonstrate that the proposed UPDeT-based multi-agent reinforcement learning achieves significant results relative to state-of-the-art approaches, demonstrating advantageous transfer capability in terms of both performance and training speed (10 times faster).

Published as a conference paper at ICLR 2021
UPDET: UNIVERSAL MULTI-AGENT REINFORCEMENT LEARNING VIA POLICY DECOUPLING WITH TRANS- FORMERS
Siyi Hu', Fengda Zhu', Xiaojun Chang", Xiaodan Liang Monash University, 2un Yat-sen University, Dark Matter Al Inc. {siyi. hu, fengda. emonash.. edu {cxj273, 328 @ggmgmma i com
ABSTRACT
Recent advances in multi-agent reinforcement learning have been largely limited training one model from scratch for every new task. This limitation occurs due to the restriction of the model architecture related to fixed input and output di- mensions, which hinder the experience accumulation and transfer of the learned agent over tasks across diverse levels of difficulty (e.g. 3 3 or 5 vs 6 multi- agent games). In this paper, we make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing single architecture to tasks with different observation and action configuration requirements. Unlike previous RNN-based models, we utilize transformer-bast bassd model generate a flexible policy by decoupling the policy distribution from the intertwined in- put observation, using an importance weight determined with the aid of the self- attention mechanism. Compared to a standard transformer block, the proposed model, which we name Universal Policy Decoupling Transformer (UPDeT), fur- ther relaxes the action restriction and makes the multi-agent task's decision pro- cess more explainable. UPDeT is general enough to be plugged into any multi- agent reinforcement learning pipeline and equip it with strong generalization abil- ities that enable multiple tasks to be handled at a time. Extensive experiments on large-scale SMAC multi-agent competitive games demonstrate that the proposed UPDeT-based multi-agent reinforcement learning achieves significant improve- ments relative to state-of-the-art approaches, demonstrating advantageous transfer capability in terms of both performance and training speed (10 times faster). Code is available https: /ggtthbt com/ hhhusip iyi monash/UPDeT"
INTRODUCTION
Reinforcement Learning (RL) provides framework for decision-making problems in interactive environment, with applications including robotics control (Hester et al. (2010)), video gaming (Mnih al. (2015)), auto-driving (Bojarski et al. (2016)), person search (Chang al. (2018)) and vision- language navigation (Zhu et al. (2020)). Cooperative multi-agent reinforcement learning (MARL), long-standing problem in the RL context, involves organizing multiple agents to achieve a goal, and is thus key tool used to address many real-world problems, such as mastering multi-player video games (Peng et al. (2017)) and studying population dynamics (Yang et (2017)).
A number of methods have been proposed that exploit an action-vall function to learn multi- agent model (Sunehag et al. (2017), Rashid et al. (2018), Du al. (2019), Mahajan et al. (2019), Hostallero et al. (2019), Zhou et al. (2020), Yang et al. (2020)). However, current methods have poor representation learning ability and fail to exploit the common structure underlying the tasks this is because they tend to treat observation from different entities in the environment of the whole. Accordingly, they give tacit support to the assumption that neural networks are able to automatically decouple the observation to find the best mapping between the whole observation and policy. Adopting this approach means that they treat all information from other agents or different parts of the environment in the same way. The most commonly used method involves concatenating
integral part
"Corresponding author.

---

Published as a conference paper at ICLR 2021
Feed Fonward Propugation
Loss Function
GRU/LSTM
Qtor O, a)
V F
Credit Assignment &
actions
Cu (a(o
Qo a
Individual g
Individual &
Transformer
0
of
Agent
T-1 ally
action action
action action
action
action 1
Ni enemy
Agent 1
Agent T
actions
Figure I: An overview of the MARL framework. Our work replaces the widely used GRU/LSTM: based individual value function with transformer-based function. Actions are separated into action groups according to observations.
the observations from each entity in to vector that is used as input (Rashid et al. (2018), Du et al. (2019), Zhou et al. (2020)). In addition, current methods ignore the rich physical meanings behind each action. Multi-agent tasks feature close relationship between the observation and output. If the model does not decouple the observation from the different agents, individual functions maybe misguided and impede the centralized value function. Worse yet, conventional models require the input and the output dimensions to be fixed (Shao et al. (2018), Wang et al. (2020)), which makes zero-shot transfer impossible. Thus, the application of current methods limited in real-world applications.
Our solution to these problems to develop a multi-agent reinforcement learning (MARL) frame- work with no limitation on input or output dimension. Moreover, this model should be general enough to be applicable to any existing MARL methods. More importantly, the model should be explainable and capable of providing further improvement for both the final performance on single- task scenarios and transfer capability on multi-task scenarios.
Inspired by the self-attention mechanism (Vaswani et al. (2017)), we propose transformer-basee MARL framework, named Universal Policy Decoupling Transformer (UPDeT), There are four key advantages of this approach: I) Once trained, it can be universally deployed; 2) it provide more robust representation with a policy decoupling strategy; 3) it is more explainable; 4) it is general enough to be applied any MARL model. We further design transformer- based function to handle various observation sizes by treating individual observations as "observation-entiiies""" We match the related observation-entity with action-groups by separating the action space into several action-groups with reference to the corresponding doservation-entiig -ennitttttttttt allowing us to get matched observation-entity - action-group pairs set. We further use a self-attention mechanism to learn the relationship between the matched observation-entily and other observation-entities. Through the use of self-attention map and the embedding of each observation-entin policy at an action- action-group level. We refer to this strategy as Policy Decoupling. By combining the transformer and policy decoupling strategies, UPDeT significantly outperforms conventional RNN- based models.
UPDeT can optimize the
In UPDET, there is no need to introduce any new parameters for new tasks. We also prove that is only with decoupled policy and matched observation-entity action-group pairs that TPDeT can learn a strong representation with high transfer capability. Finally, our proposed UPDeT can be plugged into any existing method with almost changes to the framework architecture required, while still bringing significant improvements to the final performance, especially in hard and com- plex multi-agent tasks.
The main contributions of this work are as follows: First, our UPDeT-based MARL framework outperforms RNN-based frameworks by large margin in terms of final performance on state-of-
2

---

Published as a conference paper at ICLR 2021
Transtormer
Transtormer
Transformer
(a) Vanilla
Aggregation
(c) Universal
92
Transformer
Transformer
62
AP
(d) Universal Global
(o) Universal Individual
Figure 2: Three variants on different policy decoupling method types (upper part) and two variants on different temporal unit types (bottom). 'AR', 'MA' and 'EXP' represent Action Restriction, Multi-tan bulti-tasken at A time and EXPlainable, respectively. ding, q-value and hidden states with n observation entities and available actions. G represents for the global hidden state and t is the current time step. A black circle indicates that the variant pos- sesses this attribute; moreover, variant (d) is our proposed UPDeT with best performance. Further details on all five variants can be found in Section 3.
and h represents for observation, embed-
the-art centralized functions. Second, model has strong transfer capability and handle number of different tasks at time. Third, our model accelerates the transfer learning speed (total steps cost) to make roughly 10 times faster compared RNN-based models in most scenarios.
RELATED WORK
Attention mechanisms have become integral part of models that capture global dependencies. In particular, self-attention (Parikh et al. (2016)) calculates the response at specific position sequence by attending to all positions within this sequence. Vaswani et al. (2017) demonstrated that machine translation models can achieve state-of-the-art results solely by using a self-attention model. Parmar et al. (2018) proposed Image Transformer model that applies self-attention to image generation. Wang et al. (2018) formalized self-attention as a non-local operation in order to model the spatial-temporal dependencies in video sequences. In spite of this, self-attention mecha- nisms have not yet been fully explored in multi-agent reinforcement learning.
Another line of research is multi-agent reinforcement learning (MARL). Existing work in MARL focuses primarily on building centralized function to guide the training of individual value function (Lowe et al. (2017), Sunehag al. (2017), Rashid al. (2018), Mahajan al. (2019), Hostallero et al. (2019), Yang et al. (2020), Zhou et al. (2020)). Few works have opted to form better individ- ual functions with strong representation and transfer capability. In standard reinforcement learning, this generalization has been fully studied (Taylor & Stone (2009), Ammar et al. (2012), Parisotto al. (2015), Gupta al. (2017), Da Silva Costa (2019)). While multi-agent transfer learning has been proven to be more difficult than the singlc-agent scenario (Boutsioukis et al. (2011), 1). Shao et al. (2018), Vinyals et al. (2019)). However, the transfer capability of a multi-agent system is of greater significance due the various number of agents, observations sizes and policy distributions.
To the best of our knowledge, we are the first to develop a multi-agent framework capable of handling multiple task at time. Moreover, we provide policy decoupling strategy to further improve the model performance and facilitate the multi-agent transfer learning, which is a significant step towards real world multi-agent applications.
3

---

Published as a conference paper at ICLR 2021
3 METHOD
We begin by introducing the notations and basic task settings necessary for our approach. We then describe a transformer-based individual function and policy decoupling strategy under MARL. Fi- nally, we introduce different temporal units and assimilate our Universal Policy Decoupling Trans- former (UPDeT) into Dec-POMDP.
3.1 NOTATIONS AND TASK SETTINGS
Multi-agent Reinforcement Learning A cooperative multi-agent task is decentralized par- tially observable Markov decision process (Oliehoek et al. (2016)) with a tuple G (S, A, U, P. r, Z, O, n, y). Let S denote the global state of the environment, while A represents the set of n agents and U is the action space. At each time step I, agent a E A = {1,.... n) se- U UT, which in turn causes a transition in
U, forming joint action the enviromment represented by the state transition function P(s'|s, u) S x U x S + [0. 1]. All agents share the same reward function r(s, u) S x tor. We consider partially observable scenario in which each agent makes individual observations E according the observation function (s,, a) x A - Z. Each agent has an action- observation history that conditions a stochastic policy n². creating the following joint action value: 1:: 1:30 [R St, Ur). where R2 = ~ is the discounted return.
lects an action
R, while y E [0, 1) is a discount fac-
Q (se, u,) = ::.
Centralized training with decentralized execution Centralized training with decentralized execu- tion (CTDE) a commonly used architecture in the MARL context. Each agent is conditioned only on its ootteooobservation history to make decision using the learned policy. The centralized value function provides a centralized gradient to update the individual function based on its output. Therefore, stronger individual value function benefit the centralized training.
3.2 TRANSFORMER-BE BASSD INDIVIDUAL VALCE FUNCTION
In this section, we present mathematical formulation of our transformer- based model UPDeT. We describe the calculation of the global Q-function with self-attention mechanism. First, the obser- vation O is embedded into a semantic embedding to handle the various observation space. For example, if an agent a; observes other entities (o., 0i,²) } at time step t, all observation entities embedded via embedding layer E as follows:
= (E(o,), E(o{,))
(1)
Here, i is the index of the agent, i (I. Next, the value functions (Q1, 2n) for the agents for each step are estimated as follows:
di = Q. u.).
We introduce h the temporal hidden state at the last time step f f- 1, since POMDP policy is highly dependent on the historical information. eş denotes the observation embedding, while uę is the candidate action, uf E U. 6, is the parameter that defines Q, Finally, the global Q-function Qn calculated by all individual value functions, as follows:
(*,,,, u,) F(gi..., g/)
F; is the credit assignment function for defined by d; for each agent a, as utilized in Rashid et al. (2018) and Sunehag et al. (2017). For example, in VDN, F is sum function that can be expressed as F F(gi.... ... ('.) Σi= of-
Implement Q-function with Self-attention Vaswani et al. (2017) adopts three matrices, K, Q, V representing set of keys, queries and values respectively. The attention is computed as follows:
QKT Vd V
Attention (Q, V) softmax(
where dy is scaling factor equal to the dimension of the key. In method, adopt the self- attention to learn the features and relationships from the observation entity embedding and the global temporal information. To learn the independent policy in decentralized multi-agent learning,
4

---

Published as a conference paper at ICLR 2021
0
OA
0A11
0g,1
Op.
0z11
Zz
0g,1
Og.
different observation group
Transformer
02,1
Oz,1
CA1 ... C,j
pp1
2,
6z,1
93
Q2
Q4
9x
My
FC
FC
41
4x
AA
02
21
Preserve
Aggregation
Abandon
q
42.1
9c
Policy Distribution
Observation Action- Group Pair
Figure 3: The main pipeline of our proposed UPDeT, where o, e, represent observation entity, feature embedding and Q-value of each action respectively. Three operations are adopted to avoid introducing new parameters when forming the policy distribution, namely 'preserve', 'aggregation' and "abandon'. Details can found in Section 3.3 and a real case can be found in Fig. 7.
define K, Q, and V, as the key, query and value metries for each agent ag. We further consider the query, key and value for the same matrices R. = K; Q. = V,, where E (1,..., L.} is the number of layers of the transformer. Thus, formulate our transformer follows:
R = [
Q Ky V (FQ.
(5)
R = Attention (Q', Ki VT).
where I.F represents the linear functions used to compute K, Q V. Finally we project the entity features of the last transformer layer R to the output space of the value function Q. We implement the projection using linear function P:
Qi(1 ,ei, u.) P(R,, u.).
3.3 POLICY DECOUPLING
single transformer-base individual function with self-attention mechanism still unable han- dle various required policy distribution. A flexible mapping function P in Eq, 6 is needed to deal with the various input and output dimensions and provide strong representation ability. Using the correlation between input and output, design strategy called policy decoupling, which is the key part of UPDeT.
The main idea behind the policy decoupling strategy can be summarized into three points:
No restriction on policy dimension. The output dimension of a standard trans- former block must be equal to less than the input dimension. This is unacceptable in some MARL tasks, as the action number can be larger than the entity number.
Point D:
Point 2: Ability to handle multiple tasks at a time. This requires a fixed model architecture without new parameters being introduced for new tasks. Unfortunately, if point ( is satis- fied, point 2 becomes very problematic to achieve. The difficulty lies in how to reconcile points ① and @
Point 3: Make the model more explainable. It would be preferable if we can could replace the conventional RNN based model with a more explainable policy generation structure.
Following the above three points, we propose three policy decoupling methods, namely Vanilla Transformer, Aggregation Transformer and Universal Policy Decoupling Transformer (UPDeT). The pipelines are illustrated in Fig. 2. The details of the Vanilla Transformer and Aggregation Transformer are presented the experiment section and act as our baselines. In this section, we mainly discuss the mechanism of our proposed UPDeT.
5

---

Published as a conference paper at ICLR 2021
Tasking the entity features of the last transformer layer outlined in Eq. 5, the main challenge is to build strong mapping between the features and the policy distribution. UPDeT first matches the in- put entity with the related output policy part. This correspondence is easy to find in the MARL task, as interactive action between two agents is quite common. Once we match the corresponding en- tity features and actions, we substantially reduce the burden of model learning representation using the self-attention mechanism. Moreover, considering that there might be more than one interactive actions of the matched entity feature, we separate the action space into several action groups, each of which consists several actions matched with one entity. The pipeline of this process is illustrated in the left part of Fig. 3. In the mapping function, to satisfy point I and point 2, we adopt two strategies. First, if the action-group of one entity feature contains more than one action, shared fully connected layer added to map the output to the action number dimension. Second, if one en- tity feature has no corresponding action, we abandon it, there is no danger of losing the information carried by this kind of entity feature, as the transformer has aggregated the information necessary to each output. The pipeline of UPDeT can be found in the right part of Fig. 3. With UPDeT, there no action restriction and no new parameter introduced in new scenarios. A single model can be trained on multiple tasks and deployed universally. In addition, matching the corresponding entity feature and action-group satisfies point 3 as the policy is explainable using an attention heatmap, as we will discuss in Section 4.4.
3.4 TEMPORAL UNIT STRUCTURE
Notably, however transformer-basee individual value function with policy decoupling strategy cannot handle a partial observation decision process without trajectory or history information. In Dec-POMDP (Olichock et al. (2016)), each agent chooses action according to x (r"),,,, where u and r represents for action and action-observation history respectively. In GRU and LSTM, adopt& hidden state hold the information of the action-observation history. However, the combination of transformer block and a hidden state has not yet been fully studied. In this section, we provide two approaches to handling the hidden state in UPDeT:
1) Global temporal unit treats the hidden state as an additional input of the transformer block. The process is formulated in similar way Eq. 5 with the relation: R (h/ , and (hg, e RL. Here, we ignore the subseript and instead use G represent 'global.. The global temporal unit is simple but efficient, and provides us with robust performance in most scenarios.
2) Individual temporal unit treats the hidden state as the inner part of each entity. In other words, each input maintains its own hidden state, while each output projects a new hidden state for the next time step. The individual temporal unit uses a more precise approach to controlling history information as it splits the global hidden state into individual parts. We use j to represent the number of entities. The relation of input and output is formulated as R (M hh- _- ,ef) and (hi...'', } = R4, However, this method introduces the additional burden of learning the hidden state independently for each entity. In experiment Section 4.1,2, we test both variants and discuss them further.
3.5 OPTIMIZATION
We use the standard squared TD error in DQNs (Mnih et al. (2015)) to optimize our entire frame- work as follows:
L(O) Σ DQN -
Q(s, - ())
Here, b represents the batch size. In partially observable settings, agents can benefit from condition- ing on action-observation history. Hausknecht & Stone (2015) propose Deep Recurrent Q-networks (DRQN) for this sequential decision process. For our part, replace the widely used GRU (Chung et al. (2014))LSTM (Hochreiter & Schmidhuber (1997)) unit in DRQN with a transformer-bas based temporal unit and then train the whole model.

---

Published as a conference paper at ICLR 2021
3.0
Merred
Lhieerod CRU
01.5
OPRN
0.0
VEN
Test Win Rate
Test Rats
0.6
Test Rast
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0 Steg
1.5
2.0
2.0
Stepr
Step
(a) Policy variants
(b) Temporal variants
(c) MARL methods
1.0
1.0
Smihard-)
0.9
Dm (ms)hand)
0.8
CRU
BodoMedd!
Test Win Rake
Test Whr Rota
UROIT
CRU
Test Win Rate
0.6
Scerate 3risagy
0.6
0.4
0.2
arleasy
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
3.0 Step
3.5
2.0
0.0
0.5
1.0
1.5
2.0
Step
(d) Easy scenarios
(c) Hard scenarios
(I) Mismatch experiment
Figure 4: Experimental results with different task settings. Details can be found in Section 4.1.2.
4 STARCRAFT II EXPERIMENT
In this section, we evaluate UPDeT and its variants with different policy decoupling methods in the context of challenging micromanagement games StarCraft II. We compare UPDeT with the RNN-based model on a single scenario and test the transfer capability on multiple-scenario transfer tasks. The experimental results show that UPDeT achieves significant improvement compared to the RNN-based model.
4.1 SINGLE SCENARIO
In the single scenario experiments, we evaluate the model performance on different scenarios from SMAC (Samvelyan et al. (2019)). Specifically, the scenarios considered are as follows: 3 Marines vs 3 Marines (3m, Easy), 8 Marines Hard+) and 5 Marines vs 6 Marines (5m_vs_6m, Hard). In all these games, only the units from player's side are treated as agents. Dead enemy units will masked out from the action space to ensure that the executed action is valid. More detailed settings can be acquired from the SMAC environment (Samvelyan et al. (2019)).
Marines (8m, Easy). Marines vs 5 Marines (4m vs 5m,
4.1.1 METHODS AND TRAINING DETAILS
The MARL methods for evaluation include VDN (Sunehag et al. (2017)), QMIX (Rashid et al. (2018)) and QTRAN (Hostallero et al. (2019)). All three SOTA methods' original implementation can be found at https: higgtbt.bbbgggggggggogggg..bg... com/ www.www/wwmaaa......r..
These methods were selected due to their robust performance across different multi-agent tasks. Other methods, including COMA (Foerster et al. (2017)) and IQL (Tan (1993)) do not perform stable across in all tasks, have been proved in several recent works (Rashid et al. (2018), Mahajan et al. (2019). Zhou et al. (2020)). Therefore, combined UPDeT with VDN, QMIX and QTRAN to prove that model can im- prove the model performance significantly compared to the GRU-based model.
4.1.2 RESULT
The model performance result with different policy decoupling methods be found in Fig. 4a. Vanilla Transformer is our baseline for all transformer- based models. This transformer only satis- fies point ② Each output embedding can either be projected action or abandoned. The vanilla
7

---

Published as a conference paper at ICLR 2021
1.0
1.0
0.8
Textt Win Rate
Test Win Rate Rate
0.6
0.6
0.4
7m
5m
5m
0.2
perfoiTanON
Ini Tander CRU-Transker Uni Scranch
0.2
Uni Fambler chikTranster
0.0
0.0
Uni-Scranch
0.0
0.2
0.4
D.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Step
346
Step
led
(a) Transfer from 7 marines to 3 marines
(b) Transfer from 3 marines to 7 marines
Figure 5: Experimental results on transfer learning with UPDeT (Uni-Transfer) and GRU unit (GRU-Transfer), along with UPDeT training from scratch (Uni-Scratch). At time step 0 and 500k, we load the model from the source scenario and finetune on the target scenarios. The circular points indicate the model performance on new scenarios without finetuning.
transformer fails to beat the enemies in the experiment. Aggregation Transformer is variant of vanilla transformer, the embedding of which are aggregated into global embedding and then pro- jected to a policy distribution. This transformer only satisfies the point Q. The performance of the aggregation transformer is worse than that of the GRU-based model. The result proves that it is only with policy decoupling strategy that the cransoommr-bbsed model can outperform the conventional RNN-based model. Next, we adopt UPDeT to find the best temporal unit architecture in Fig. 4b. The result shows that without hidden state, the performance is significantly decreased. The tempo- ral unit with global hidden state more efficient terms of convergence speed than the individual hidden state. However, the final performances model, we combine the UPDeT with VDN QMIX / QTRAN respectively and compare the final performance with RNN-based methods in Fig. 4c. We evaluate the model performance 5m_vs_6m (Hard) scenarios. Combined with UP all three MARL methods obtain significant improvement by large margins compared to the GRU-based model. The result proves that our model can be in- jected into any existing stat- of-the-art MARL method to yield better performance. Further more, we combine UPDeT with VDN and evaluate the model performance on different scenarios from Easy to Hard+ in Fig. 4d and Fig. 4e. The results show that the UPDeT performs stably on easy scenarios and significantly outperforms the GRU-based model on hard scenarios, in the 4m vs. 5m(Hard++) scenario, the performance improvement achieved by UPDeT relative to the GRU-based model is of the magnitude of around 80%. Finally, we conduct an ablation study on UPDeT with paired and un- paired cntervattoo-enttt--aeeeeeee action-group. the result of which are presented in Fig. 4f. We disrupt the original correspondence between 'attack' action and enemy unit. The final performance is heavily decreased compared to the original model, and is even worse than the GRU-based model. We ac- cordingly conclude that only with policy decoupling and a paired observation-entity -aatonenttttttttttnnnnnaeee action-group strategy can UPDeT learn strong policy.
almost the same. To test the generalization of our
4.2 MULTIPLE SCENARIOS
In this section, we discuss the transfer capability of UPDeT compared to the RNN-based model. We evaluate the model performance in a curriculum style. First, the model is trained one the 3m (3 Marines vs 3 Marines) scenario. We then used the pretrained 3m model to continually train on the 5m (5 Marines Marines) and 7m (7 Marines vs Marines) scenarios. We also conduct experiment in reverse from 7m to 3m. During transfer learning, the model architecture of UPDeT remains fixed. Considering that the RNN-based model cannot handle various input and output dimensions, we modify the architecture of the source RNN model when training on the target scenario. We preserve the parameters the GRU cell and initialize the fully connected layer with proper input and output dimensions to fit the new scenario. The final results can be seen in Fig. 5a and Fig. 5b. Our proposed UPDeT achieves significantly better results than the GRU-based model. Statistically, UPDeT'ss total timestep cost to converge is least 10 times less than the GRU-basee model and 100 times less than training from seratch. Moreover, the model demonstrates a strong generalization ability without finetuning, indicating that UPDeT' learns robust policy with meta-level skill.
8

---

Published as a conference paper at ICLR 2021
4.3 EXTENSIVE EXPERIMENT ON LARGE-SCALL MAS
To evaluate the model performance in large-scale scenarios, we test our proposed UPDeT on the 10m_vs_1 Im and 20m_vs.21m, scenarios from SMAC and 64_vs_66 battle game in the MAgent Environment (Zheng et al. (2017)). The final results can be found in Appendix E.
4.4 ATTENTION BASED STRATEGY: AN ANALYSIS
The significant performance improvement achieved by UPDeT on the SMAC multi-agent challenge be credited the self-attention mechanism brought by both transformer blocks and the policy decoupling strategy in UPDeT. In this section, we mainly discuss how the attention mechanism assists in learning much more robust and explainable strategy. Here, we use the Marines vs Marines game (therefore, the size of the raw attention matrix is 6x6) as an example to demonstrate how the attention mechanism works. As mentioned in the caption of Fig. 6, we simplify the raw complete attention matrix to grouped attention matrix. Fig, 6b presents the three different stages one episode including Game Start, Attack and Survive, with their corresponding attention matrix and strategies. In the Game Sturf stage, the highest attention is in line col 3 of the matrix, indicating that the agent pays more attention to its allies than its enemies. This phenomenon can be interpreted follows: in the startup stage of one game, all the allies are spawned at the left side of the map and are encouraged to find and attack the enemies on the right side In the Attack stage, the highest attention is in line 2 col 2 of the matrix, which indicates that the enemy is now in the agent's attack range; therefore, the agent will attack the enemy to get more rewards. Surprisingly, the agent chooses to attack the enemy with the lowest health value. This indicates that long term plan can be learned based on the attention mechanism, since killing the weakest enemy first can decrease the punishment from the future enemy attacks. In the Survive stage, the agent's 's health value is low, meaning that it needs to avoid being attacked. The highest attention is located in line I col 1, which clearly shows that the most important thing under the current circumstances is to stay alive. For as long as the agent is alive, there is still chance for it to return to the front line and get more reward while enemies are attacking the allies instead of the agent itself.
In conclusion, the self-attention mechanism and policy decoupling strategy of UPDeT provides a strong and clear relation between attention weights and final strategies. This relation can help us better understand the policy generation based the distribution of attention among different entities. An interesting idea presents itself here: namely, if we can find a strong mapping between attention matrix and final policy, the character of the agent could be modified in unsupervised manner.
5 CONCLUSION
In this paper, we propose UPDeT, a universal policy decoupling transformer model that extends MARL to much broader scenario. UPDeT is general enough to be plugged into any existing MARL method. Moreover, our experimental results show that, when combined with UPDeT, ex- isting state-of-the-ee
MARL methods can achieve further significant improvements with the same training pipeline. On transfer learning tasks, our model 100 times faster than training from scratch and 10 times faster than training using the RNN-based model. In the future, we aim to develop centralized function based on UPDeT and apply the self-attention mechanism to the entire pipeline of MARL framework to yield further improvement.
ACKNOWLEDGGMENTS
This work was supported in part by the National Natural Science Foundation of China (NSFC) un- der Grant No. No.U1992073 and in part by the National Natural Science Foundation China (NSFC) under Grant No. 619762333333 and No.6190611 and Australian Research Council Discovery Early Ca- reer Researcher Award (DE190100626), and Funding "Leading Innovation Team of the Zhejiang Province" 2001801017)..

---

Published as a conference paper at ICLR 2021
REFERENCES
Haitham B Ammar, Karl Tuyls, Matthew E Taylor, Kurt Driessens, and Gerhard Weiss. Reinforce- ment learning transfer via sparse coding. In Proceedings of the / 11th international conference on autonomous agents and multiagent systems, volume 1, pp. 383-390. International Foundation for Autonomous Agents and Multiagent Systems.... 2012.
Mariusz: Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End end learning for self-driving cars. arXiv preprint 1604iv:6004.733 2016.
Georgios Boutsioukis, Ioannis Partalas, and Ioannis Vlahavas. Transfer learning in multi-agent reinforcement learning domains. European Workshop on Reinforcement Learning, pp. 249- 260. Springer, 2011.
Xiaojun Chang, Po- Yao Huang, Yi-Dong Shen, Xiaodan Liang, Yi Yang, and Alexander G. Haupt- mann. RCAA: relational context-aware agents for person search. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IX, 2018.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Felipe Leno Da Silva and Anna Helena Reali Costa. A survey transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.
Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individ- ual intrinsic reward multi-agent reinforcement learning. Advances in Neural Information Processing Systems, pp. 4403-4414, 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:/705.000 2017.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv: 1703.02999, 2017.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint 1oXXv:1507.062777 2015.
Todd Hester, Michael Quinlan, and Peter Stone. Generalized model learning for reinforcement learn- ing on humanoid robot. In 2010 IEEE International Conference on Robotics and Automation, pp. 2369-2374. IEEE, 2010.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1785-1780, 1997.
Wan Ju Kang David Earl Hostallero, Kyunghwan Son, Daewoo Kim, and Yung Yi Qtran. Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the 31st International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR. 2019.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperativecompettivve environments. In Advances in neural infor- mation processing systems, pp. 6379-6390, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7613-7624,
2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, (177550):559-5333 2015.
10

---

Published as a conference paper at ICLR 2021
Frans A Oliehoek, Christopher Amato, al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit, A decomposable attention model for natural language inference. arXiv preprint arXiv: 01933, 2016.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.0644 2015.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv: arXiv:/802.05551, 2018.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang- Multiagent
nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint 1703. 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1485, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 190.:09902.00444 2019.
Kun Shao, Yuanheng Zhu, and Dongbin Zhao. Starcraft micromanagement with reinforcement learning and curriculum transfer learning. IEEE Transactions Emerging Topics in Computa- tional Intelligence, 3(1):73-84 84,,,4442 2018.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. deloppseconooiiioonn networks for cooperative multi-agent learning. arXiv preprint 17Xxv:1706.05266, 2017.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev. et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)330--334 2019.
Weixun Wang, Tiappei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In AAAI, pp. 7293-7300, 2020.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Yaodong Yang, Lantao Yu, Yiwei Bai, Jun Wang, Weinan Zhang, Ying Wen, and Yong Yu. A study of ai population dynamics with million-agent reinforcement learning. arXiv preprint arXiv:1709.04511, 2017.
Yaodong Yang, Rui Luo, Minne Li. Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. arXiv preprint arXiv:1802.05438. 2018.
Yaodong Yang, Ying Wen, Lihuan Chen, Jun Wang, Kun Shao. David Mguni, and Weinan Zhang. Multi-agent determinantal l-laarning. arXiv preprint arXiv:2006.01482, 2020.
11

---

Published as a conference paper at ICLR 2021
Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. arXiv preprins arXi::1712.00600, 2017.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for multi-agent actor-critic. arXiv preprint arXiv:2007.022. 2020.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition, pp. 10012-10022, 2020.
12

---

Published as a conference paper at ICLR 2021
A DETAILS OF SMAC ENVIRONMENT
The action space contains four movement directions, k attack actions (where k the fixed maximum number of the enemy units map). stop and none-operation. each time step, the agents receive joint team reward, which is defined by the total damage incurred by the agents and the total damage from the enemy side. Each agent is described by several attributes, including health point HP, weapon cool down (CD), unit type, last action and the relative distance of the observed units. The enemy units are described in the same way except that CD excluded. The partial observation of agent comprises the attributes of the units, including both the agents and the enemy units, that exist within its view range, which a circle with a specific radius.
B DETAILS OF MODEL
The transformer block in all different experiments consists of heads and 2 layer transformer blocks. The other important training hyper parameters are as follows:
List of Hyper Parameters
Name
Value
batch size
32
test interval
2000
gamma
0.99
buffer size
5000
token dimension (UPDeT))
32
channel dimension (UPDeT)
32
epsilon start
1.0
epsilon end mn hidden dimension
0.05
64
target net update interval
200
mixing embeddding dimension (QMIX)
32
hypernet layers (QMIX)
2
hypernet embedding (QMIX)
64
mixing embeddding dimension (QTRAN)
32
opt loss (QTRAN)
1
nopt min loss (QTRAN)
0.1
C
SOTA MARL VALUE-BASED FRAMEWORK
The three SOTA method can be briefly summarized as follows:
VDN (Sunehag al. (2017)): this method learns an individual Q-value function and rep- resents Qtot a sum of individual Q-value functions that condition only on individual observations and actions.
QMIX (Rashid (2018)): this method learns decentralized Q-function for each agent, with the assumption that the centralized Q-value increases monotonically with the individ- ual Q-values.
QTRAN (Hostallero al. (2019)): this method formulates multi-agent learning as an opti- mization problem with linear constraints and relaxes it with L2 penalties for tractability.
D UPDET ON SMAC: A REAL CASE
We take the 3 Marines vs 3 Marines challenge from SMAC with UPDeT example; more details can be found in Fig. 7. The observation are separated into groups: main agent, two other ally agents and three enemies. The policy output includes basic action corresponding to the main agent's 's observation and attack actions, one for each enemy observation. The hidden state is added after the embedding layer. The output of other agents is abandoned there corresponding
13

---

Published as a conference paper at ICLR 2021
sterter
exery
complete ettestion
Current Stage
Stage 1. Came Start
Stage 2 Attacs
Stage S Survive
HAA Low
trodeeee erano ai
[move]
[now]]
(move]
Attention
(nttack)
(etacc
(rsack)
(void)
[yokd]]
(vikd]
frod
attrtion
(nsove)
[attac]]
(nove]
bost
Stratogy
locass on idlie move (with allies
focu: arimie sraack Jone enemy)
Escus denns health move tO SuRViNe
(a) Attention Matrix
(b) Strategy with Attention
Figure 6: An analysis of the attention based strategy of UPDeT. Part (a) visualizes typical attention matrix. Part (b) utilizes the simplified attention matrix to describe the relationship between attention and final strategy. Further discussion can be found in Section 4.4.
action. Once agent enemy has died, we mask corresponding unavailable action in the action select stage to ensure only the available actions are selected.
E RESULTS OF EXTENSIVE EXPERIMENT ON LARGE SCALE
We further test the robustness of UPDeT in large-scale multi-agen system. To do so, enlarge the game size in SMAC (Samvelyan et al. (2019)) to incorporate more agents and enemies on the battle field. We compare the performance between the UPDeT and GRU-based approaches. In the 20 Marines vs 21 Marines game, accelerate the training and satisfy the hardware limitations, we decrease the batch size of both the GRU baseline and UPDeT from 32 to 24 in the training stage. The final results can be found in Fig. 8a. The improvement is still significant in terms of both sample efficiency and final performance. Moreover, it is also worth mentioning that the model size of UPDeT stays fixed, while the GRU-based model becomes larger in large-scale scenarios. In the 20 Marines vs 21 Marines game, the model size of GRU is almost double that of UPDeT. This indicates that UPDeT is able to ensure the lightness of the model while still maintaining good performance.
10 Marines 11 Marines game and 20 Marines 21 Marines game to
We also test the model performance in the MAgent Environment (Zheng et al. (2017)). The set- tings of MAgent are quite different from those of SMAC. First, the observation size and number of available actions not related the number of agents. Second, the 64 64 battle game tested is a two-player zero-sum game which is another hot research area that combines both MARL and GT (Game Theory), the most successful attempt in this area involves adopting mean-field approximation of GT in MARL to accelerate the self-play training (Yang et al. (2018)). Third, as for the model architecture, there is no need to use recurrent network like GRU in MAgent and the
agent
enemy
enemy 2 enemy
ally
ally
Raw
Transformer
moan
00.0
Embedding to Policy
C
abandon part
siua
nJ Ug
alackk allack allack enerry anamy enemy
Q value embedding
1
2
Figure 7: Real case on 3 Marines Marines Challenge from SMAC.
14

---

Published as a conference paper at ICLR 2021
100
Scanario
CON-beseline-m
Ys
200
DON-trenafommer-in
0.8
20m, 21m
BaseModel
Test Win Rate
0.6
GRU
Reward
-200
-400
-500
0.0
0.00
0.25
0.50
0.75
1,00
2.25
1.50
1:75
2,00
230
200
130
1020
1220
1500
1750
2000
stop
120
(a) Large-Scale Scenarios
(b) MAgent Battle: vs
Figure 8: Experimental results on the large-scale MAS, including SMAC and MAgent.
large observation size requires the use of CNN from embedding. However, ny treating TPDeT as pure encoder without recurrent architecture, we can still conduct experiments on MAgent; the final results of these can be found in Fig. 8b. As the result show, UPDeT performs better than the DQN baseline, although this improvement is not as significant as it in SMAC.
15


Reviews:
[{"review_id": "wZpIlaSEAGi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Summary and claims of the paper\n\nThe authors propose several Transformer-based architectures that can be used in combination with a MARL (multi-agent reinforcement learning) algorithm to tackle multi-agent environments. They identify a particularly suitable architecture they call UPDeT which they claim combines the following advantages:\n 1) Can be used in the multi-task setting, where the number of entities and therefore the inputs changes from one task to the next.\n 2) Can handle a varying number of actions per entity (e.g. the controlled units could have N actions and each enemy unit corresponds to 1 action (which one to target), allied units correspond to no actions).\n 3) Is more explainable because the attention weights of the transformer can be inspected to see which information a given output depends on.\n 4) Outperforms the standard RNN approach on specific tasks.\n 5) Contrary to the standard RNN approach, the architecture is able to zero-shot generalize to tasks with different numbers of entities and learns significantly faster than the RNN architecture when fine-tuning on a new task. (This is in my opinion the most important and interesting claim).\n\nI have collected these claims from different parts of the paper and had to add some of my own interpretation to disambiguate what was meant. Please correct me if I misunderstood any of the claims. It would be nice if all of these were stated clearly in one place in the paper.\n\nPoints 1-3 can be argued based on the design of the architecture alone, while points 4) and 5) are supported by experiments using the SMAC (StarCraft Multi-Agent Challenge) environment.\n\nA summary of UPDeT from my understanding:\nIn MARL a group of independently acting agents needs to be trained to maximize a reward. Each agent has a separate set of Q values for each action that it can issue at a given time. In UPDeT, the Q values for one agent are computed by applying a transformer to all of the entities that the given agent can see. The resulting transformer output is then mapped into Q values depending on the positions in the transformer output. E.g. in the case of SMAC the position of the currently controlled unit corresponds to many actions, like moving left/right/up/down. The position of enemy units corresponds to 1 action each (targeting that unit) and the position of allied units (also controlled by the network, but not by this application of the transformer) corresponds to 0 actions. The mapping from transformer output to N actions is handled through a linear layer that is shared between entity groups (e.g. outputs for all enemy units get transformed with the same linear layer). A recurrent state through time can be added either globally for the whole transformer, or added to the individual units.\n\n### Discussion of prior work\n\nThe section on prior work is split between references to transformer and MARL literature.\nFrom what I can tell it is complete enough to position the paper sufficiently in the literature.\nThe one thing that I think is missing is a reference to the AlphaStar paper (https://www.nature.com/articles/s41586-019-1724-z) since this is an example of an agent that uses a transformer and is also applied to StarCraft.\nIn contrast to this work it doesn't use a MARL algorithm to control multiple units.\n\n### Are the claims supported?\n\nClaims 1) and 2) are supported directly by the design of the architecture.\nI struggled to understand the details of the architecture for some time. In my opinion this part of the paper could really benefit from a clearer presentation of the architecture (more details and some suggestions for improvement are in the next section).\n\nI'm somewhat unsatisfied with how claim 3) is handled in the paper, because the authors don't include any demonstration of explainability (e.g. a particular instance where the attention weights make it clear how a certain decision is made). Simply claiming that the architecture is explainable because a transformer is used is not enough, because in my experience there can be plenty of instances where we can't interpret how a given decision was made through the attention weights. It would be good if the authors would either give an interpretable example, or soften their claims of explainability and remove it from the main set of claims.\n\nClaims 4) and 5) are handled pretty well in my opinion.\nIn my experience the SMAC challenges are decently complex and allow meaningful differences in performance between algorithms to be measured.\nThe results in figure 4 support the design decisions in the architecture and show that the proposed architecture outperforms the standard RNN approach.\nThe only thing I'm unsure about here is the \"vanilla transformer\" architecture, which doesn't seem to be working at all. Is it possible that the transformer output is not conditioned on the different action types? E.g. this could be achieved by adding a positional encoding to the transformer corresponding to the different actions that can be issued. In any case such an architecture doesn't make a lot of sense since the actions are not related to the entities processed by the transformer. Maybe the aggregation transformer should be the \"vanilla\" version and the vanilla version can just be removed to make the paper simpler?\n\nClaim 5) is supported by a separate set of experiments where the agents are trained with a given number of units, and then switched to a task with a different number of units. UPDeT still performs reasonably well after the switch (zero-shot generalization) and quickly recovers to full performance during fine-tuning. I found this result genuinely interesting and I like how it is presented in the paper, e.g. by including the from-scratch curves as well.\nAs pointed out in the paper, the GRU baseline had to have some of its weights reset when switching from one task to another due to the changed action space. I wonder what would happen if a similar technique to UPDeT was used on the GRU, e.g. by emitting actions from different action heads depending on the identity of a unit as the GRU is unrolled (ideally it should be bidirectional). That way the resetting could be avoided and a fairer comparison between RNN and transformer could be made. This is not a crucial addition to the paper, but could be interesting.\n\n### Presentation and clarity\n\nI think the presentation and clarity of the paper should be improved significantly.\nThere are currently many typos, grammatical mistakes and missing words that made it hard for me to understand the paper.\nThe sections 3.2, 3.3 and 3.4 in particular are really crucial for understanding the paper and have some mistakes that confused me significantly.\nFor example in the bottom of formula (5) the attention is applied to three identical variables $R_i^l$ (this is done in other places in the paper as well, I'm not sure how to interpret it).\nIn Figure 3 the embedding outputs are labeled $e_{B,1}\\cdots o_{B,k}$, but I think this should be $e_{B,1}\\cdots e_{B,k}$.\nI found Figures 2 and 3 very difficult to understand, maybe because they show a very general version of the idea rather than the specific one that was tested by the authors.\nThe most confusing part of Figure 2 for me was the alternate white/colored filling scheme on the various tensors. What does it mean for a position to be colorless or colored here?\nI found Figure 6 in the appendix extremely useful for understanding the architecture. Maybe this should be part of the main text?\nI can appreciate that the authors want to present a general view of their idea (a \"framework\" rather than a specific architecture), but personally I would have understood the paper faster if it would focus on the specific case used for SMAC.\nThere are some references to \"training speed\" in the paper. It took me some time to figure out that this refers to the number of update steps rather than the actual speed of training the network. It would be good to clarify that.\n\nOverall I think the impact of the paper could really benefit from improving presentation and clarity.\nI have not taken this aspect into account in my rating.\n\n### Conclusion\n\nPros:\n - Benefit of using a transformer in a MARL setup is demonstrated in a non-trivial environment.\n - The proposed architecture goes beyond \"obvious\" approaches like the pooling the output of the transformer and the benefit of this is demonstrated.\n\nCons:\n- The paper is currently difficult to understand in several important places.\n- I think the use of transformers as presented here is only valid for environments with structured observations, e.g. where actions map nicely to specific entities in the observation.\n\nOverall I think the paper presents a promising approach to building MARL architectures. The approach makes sense and the experiments are insightful, but the presentation of the paper needs to be improved.\n\n*Edits after author comments and revision:*\nThe authors have greatly improved the readability of the paper. I also appreciate the addition of section 4.4. which seems like a reasonable attempt at supporting the increased interpretability of the architecture. I feel that the paper has improved, but it wasn't quite enough for me to raise the rating from 7 to 8, so I'm leaving it at \"Good paper\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "pP5ISHhmCuu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. In this paper the authors proposed a transferrable framework for multi-agent RL, which enables the learned policies easily generalize to more challenging scenarios. This seems to be a good contribution to the community of multi-agent RL. It bears a potential to handle large-scale tasks with only limited training data, while also demonstrates more explanable policies.\n2. However, the experiments seem to be insufficient. The authors only investigate scenarios for 3 vs. 3, 5 vs. 7, which are still the easiest cases in the StarCraft II combat tasks. I suggest the authors to try more on 20 vs. 30 StarCraft combat task or more challenging scenarios, or the hundres or thousands levels of multi-agent tasks like that provided by MAgent environment[1]. And a comprehensive comparison with a similar work [2] following the curriculum learning pipeline is also worth a trial. This will make this work a strong one.\n3. A more profound analysis is needed for the experiment part. Besides the performance gains, insightful understanding of how the designed model works is also necessary.\n\n[1] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang. Mean Field Multi-Agent Reinforcement Learning. ICML 2018.\n[2] Kun Shao, Yuanheng Zhu, Dongbin Zhao. StarCraft Micromanagement With Reinforcement Learning and Curriculum Transfer Learning. IEEE Transactions on Emerging Topics in Computational Intelligence, 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NgPR_0a3iAy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In my opinion, this paper is a very good paper: novel in the approach, high impacts in both theoretic sense and practical sense, and well-written. The problem of universal multi-agent reinforcement learning for multiple tasks is very interesting and challenging, and the methodology proposed in this paper is inspiring and the demonstrated experimental results are very impressive. \n\nMain contributions of this paper are as follows, which are very significant and impressive:\n [1] The proposed UPDeT-based MARL framework outperforms RNN-based frameworks on state-of-the-art centralized functions by a large margin in terms of final performance.\n[2] The proposed model has strong transfer capability and can handle a number of different task at a time.\n[3] The proposed model accelerates the transfer learning speed so that it is about 10 times faster compared to RNN-based models in most scenarios.\n\nThe paper presentation is in good quality, the concepts and the methodologies were explained clearly. The provided experiments section is also convincing and somewhat extensive, described and presented very clearly, which supports the main claimed contribution very well.\n\nThe authors also point out some promising future research direction on top of this work, which is also helpful.\n\nOverall, I strongly support this paper to be accepted and published in ICLR. The experiment results are impressive, and the proposed methodologies are inspiring and novel. I believe that many people in the research community would find it valuable/inspiring and benefit from this paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers all agree on acceptance for this paper. The initial issues with clarity seem to have been addressed by the authors.

The paper introduces a new transformer-based architecture for MARL that enables variable input and output sizes, which is used to train the agent in a more general setting and on more diverse tasks for multi-task training. The method also produces more interpretable agents.
The paper shows results on the Starcraft multi-agent challenge (not the full game of Starcraft, but still a recognised and widely used multi-agent benchmark). The method produces solid results both in terms of final training performance and zero-shot generalisation.

Although reviewers are generally supportive of this paper, they mention that the Starcraft challenge used is somewhat simple (only few units used), and that the transformer-based architecture may not be applied to domain which lack the proper structure. 


Decision: Accept (Spotlight)