MaskMA: Towards Zero-Shot Multi-Agent Decision Making with Mask-Based Collaborative Learning
Authors: Jie Liu
Yinmin Zhang
Chuming Li
Zhiyuan You
Zhanhui Zhou
Chao Yang
Yaodong Yang
Yu Liu
Wanli Ouyang
Venue: Accepted by TMLR ()

Abstract: Building a single generalist agent with strong zero-shot capability has recently sparked significant advancements. However, extending this capability to multi-agent decision making scenarios presents challenges. Most current works struggle with zero-shot transfer, due to two challenges particular to the multi-agent settings: (a) a mismatch between centralized training and decentralized execution; and (b) difficulties in creating generalizable representations across diverse tasks due to varying agent numbers and action spaces. To overcome these challenges, we propose a Mask-Based collaborative learning framework for Multi-Agent decision making (MaskMA). Firstly, we randomly mask part of the units and collaboratively learn the policies of unmasked units to handle the mismatch. In addition, MaskMA integrates a generalizable action representation by dividing the action space into intrinsic actions solely related to the unit itself and interactive actions involving interactions with other units. This flexibility allows MaskMA to tackle tasks with varying agent numbers and thus different action spaces. Extensive experiments in SMAC reveal MaskMA, with a single model trained on 11 training maps, can achieve an impressive 77.8% average zero-shot win rate on 60 unseen test maps by decentralized execution, while also performing effectively on other types of downstream tasks (e.g., varied policies collaboration, ally malfunction, and ad hoc team play).

Published in Transactions on Machine Learning Research (09/2024)
MaskMA: Towards Zero-Shot Multi-Agent Decision Making with Mask-Based Collaborative Learning
Jie Liu Yinmin Zhang Chuming Li Zhiyuan You', Zhanhui Zhou?, Chao Yang², Yaodong Yang! Yu Liu Wanli Ouyang 1.3
Equal contribution, Corresponding author. Contact: jieliue link. culk. hk, yaodong. yang @lu..edu.cn Multimedia Laboratory, The Chinese University of Hong Kong.
The University of Sydney.
3 Shanghai Artificial Intelligence Laboratory.
Institute for AI, Peking University.
SenseTime Research.
Reviewed on
Abstract
Building single generalist agent with strong zero-shot capability has recently sparked significant advancements. However, extending this capability to multi-agent decision making scenarios presents challenges. Most current works struggle with zero-shot transfer, due to two challenges particular to the multi-agent settings: (a) mismatch between centralized training and decentralized execution; and (b) difficulties in creating generalizable representations across diverse tasks due to varying agent numbers and action spaces. To overcome these challenges, propose a Mask-Based collaborative learning framework for Multi-Agent decision making (MaskMA). Firstly, we randomly mask part of the units and collaboratively learn the policies of unmasked units to handle the mismatch. In addition, MaskMA integrates a generalizable action representation by dividing the action space into intrinsic actions solely related to the unit itself and interactive actions involving interactions with other units. This flexibility allows MaskMA to tackle tasks with varying agent numbers and thus different action spaces. Extensive experiments in SMAC reveal MaskMA, with single model trained on 11 training maps, can achieve an impressive 77.8% average zero-shot win rate on 60 unseen test maps by decentralized execution, while also performing effectively on other types of downstream tasks (e.g., varied policies collaboration, ally malfunction, and ad hoc team play).
1 Introduction
The powerful transformer-base (Vaswani et al., 2017) generalist models (Ouyang et al., 2022; Touvron et al., 2023; Brown et al., 2020; Ramesh et al., 2022) have brought artificial intelligence techniques to the daily life of people. The reinforcement learning community has also witnessed a growing amount of successes in transformer-base models (Chen et al., 2021; Carroll et al., 2022; Liu et al., 2022; Janner et al., 2022; Meng et al., 2023). However, most of these existing generalist models either fail to complete unseen tasks effectively or fail to incorporate the multi-agent A natural follow-up question is: how can one build generalist model for multi-agent
reality of decision-maker decision-makin that is capable of zero-shot transfer to new tasks (e.g., different maps and varying numbers of agents in SMAC (Samvelyan et al., 2019)).
Compared to single-agent scenarios, directly utilizing transformers for centralized training in multi-agent settings encounters two primary challenges. (a) A mismatch between centralized training and decentralized execution. Multi- agent decision-making typically follows centralized training with a decentralized execution (Gronauer & Diepold, 2022) approach. Most existing methods (Wen et al., 2022b; Tseng et al., 2022b) treat multi-agent decision-making as sequence modeling problem and directly employ transformer architectures. However, the centralized training phase of transformers utilizing all units as inputs mismatches with the decentralized execution phase where each agent's perception is limited to only nearby units. This mismatch significantly reduces performance. (b) Generalization

---

Published in Transactions on Machine Learning Research (09/2024)
MaskMA
MADT
75
Win Rale
0
Training Maps
Testing Maps
Map Name
Figure I: Win rate on training and testing maps. The blue line separates the 11 training maps on the left from the 60 testing maps the right, where the performance the testing maps is the zero-shot performance. The orange line demonstrates the substantial performance advantage of MaskMA over MADT Multi-Agent Decision Transformer (Meng et al., 2023)).
problems caused by varying numbers of agents and actions. Downstream tasks have different numbers of agents, resulting in varying action spaces. Meng et al. (2023) takes large action space and mutes the unavailable actions using an action mask to handle the variable action spaces. However, this method suffers from poor generalization because the same component of the action vector represents different physical meanings in tasks with different numbers of agents.
To address these challenges, we propese introduce two scalable techniques: a Mask-based Training Strategy (MTS) and Generalizable Action Representation (GAR). The two techniques form the basis of new mask-based collaborative learning framework for multi-agent decision-make making, named MaskMA. To address the mismatch challenge, we incorporate random masking into the attention matrix of the transformer, effectively reconciling the discrepancy between centralized training and partial observations in decentralized execution, thus bolstering the model's generalization capabilities. To handle the generalization challenge. MaskMA integrates GAR by categorizing actions into two types: intrinsic actions, which are solely related to the unit itself, and interactive actions, which involve interactions with other units. GAR predicts one unit's 's interactive action from its own features together with other units' features, instead of only relying on its own features, allowing MaskMA to generalize to diverse tasks with varying agent numbers and action spaces.
We evaluate MaskMA's performance using the StarCraft Multi-Agent Challenge (SMAC) benchmark. To validate the zero-shot performance, we evaluate the win rate in challenging setting, using only 11 maps for training and 60 unseen maps for testing. Extensive experiments demonstrate that our model significantly outperforms the previous state-of- the-art methods in zero-shot scenarios. We also introduce various downstream tasks to further verify the strong robust generalization ability of MaskMA, including varied policies collaboration, ally malfunction, and ad hoc team play. MaskMA is the first approach that achieves strong notable zero-shot capability for multi-agent maiisiinmmmiinngg (e.g., 78% win rate in SMAC). We hope this work lays the groundwork for further advancements in multi-agent fundamental models, with potential applications across wide range of domains. Our main contributions summarized three folds:
We introduce novel mask-based collaborative learning framework, MaskMA, for multi-agent decision-making. This framework pretrains a transformer architecture with Mask-based Training Strategy (MTS) and Generalizable Action Representation (GAR).
We set up challenging zero-shot setting for general models in multi-agent scenarios based on the SMAC, i.e., training on only 11 maps and testing on 60 different maps and three downstream tasks including varied policies collaboration, ally malfunction, and ad hoc team play.
To our best knowledge, MaskMA is the first general model for multi-agent decision-makiin with notable zero-shot performance. With only 11 training maps, our MaskMA has achieved an impressive average zero-shot win rate of 77.8% on 60 unseen test maps, representing 78% improvement relative to the baseline method.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2
Related Work
Masked Training in Single-agent Decision Making. DT (Chen al., 2021) casts the reinforcement learning sequence modeling problem conditioned on retum-to-go, using transformer to generate optimal action. MaskDP (Liu et al., 2022) utilizes autoencoders on state-action trajectories, learning the environment's 's dynamics by masking and reconstructing states and actions. Uni[MASK] (Carroll et al., 2022) expresses various tasks as distinct masking schemes in sequence modeling, using a single model trained with randomly sampled maskings. In this paper, we explore the design of sequences in multi-agent decision making and how it can be made compatible with the mask-based training strategy.
Multi-agent Decision Making as Sequence Modeling. MADT (Meng et al., 2023) introduces Decision Trans- former (Chen et al., 2021) into multi-agent reinforcement learning, significantly improving sample efficiency and achieving strong performance in SMAC. MAT (Wen et al., 2022a) leverages an encoder-decoder architecture, incorpo- rating the multi-agent advantage decomposition theorem to reduce the joint policy search problem into sequential decision-making process. Tseng et al. (2022a) utilize the Transformer architecture and propose method that identifies and recombines optimal behaviors through teacher policy. ODIS (Zhang et al., 2023) trains state encoder and an action decoder to extract task-invariant coordination skills from offline multi-task data. In contrast, our proposed MaskMA adapts the Transformer architecture to multi-agent decision making by designing a sequence of inputs and outputs for a generalizable action representation. This approach offers broad generalizability across varying actions and various downstream tasks.
Action Representation. Recent works have explored semantic action in decision- makinn, especially in multi-agent environments. ASN (Wang et al., 2020) focuses on modeling the effects of actions by encoding the semanties of actions to understand the consequences of agent actions and improve coordination among agents. UPDeT (Hu et al., 2021) employs policy decoupling mechanism that separates the learning of local policies for individual agents from the coordination among agents using transformers. Yang et al. (2023) proposes graph network to handle variable action spaces, applying reinforcement learning to improve finite element (Bremner, 2008) methods. Our MaskMA emphasizes sequence modeling, masking strategies, and generalizable action representation. which is similar to Yang et al. (2023), to train a generalist model in the multi-agent field. This model can be applied to unseen maps and a wide range of downstream tasks, involving varied policy collaboration, ally malfunction, and ad hoc team play.
3 Method
In this section, introduce our proposed mask-based collaborative learning framework. We start by providing formulation of the multi-agent decision- making problem. Then, introduce the Mask- Training Strategy (MTS) and the Generalizable Action Representation (GAR).
3.1
Formulation
3.1.1
Multi-Agent Preliminaries
In multi-agent scenarios, states are only partially observable, i.e., each agent has only limited sight and can only observe part of other units (e.g., enemies). Therefore, a cooperative multi-agent task is generally defined as decentralized partially observable Markov decision process (Oliehoek & Amato, 2015), denoted as G =< S, U, A, P, 0, > Here S is the global state space, and U {u is the set of N units, where the first M units are controllable and the rest N .M units are uncontrollable, often subsumed as part of the environment. A M A is the action space for controllable units. At time step t, each agent joint action a E A. The joint action a at state s E S triggers transition of G, subject to the transition function (s') 8, a. S x A x S + [o, 1J. Meanwhile, a shared reward function r (8, a) S x A + R is employed, with E [o, 1] denoting the discount factor.
MM selects action a, A. forming
The global state at the t-th time step is defined as s" = (s}, ..., s'y); where each of exclusively represents the state of unit u4, and does not include any information about other units. The local observation of for each unit u, is composed of the states of all units visible the 1-th step, expressed of {of | i p!), where of indicates the indexes of
3

---

Published in Transactions on Machine Learning Research (09/2024)
Training
MaskMA
Testing
GAR
Zera shat
M
FC
Transformer
Varied Palicies
A
Module
Hidden
Generalizable Representation
Controliable
Input
Network Flow
Teammate Malfunction Ad hoc Play
Action
Concatenation
Mask-based Training Strategy
Figure 2: MaskMA employs the transformer architecture combined with generalizable action representation and then trained through mask-based training strategy. It effectively generalizes skills and knowledge from training maps into various downstream tasks, including unseen maps, varied policies collaboration, ally malfunction, and ad hoc team play.
units within u,'s 's observation range. Actions are categorized into two types: intrinsic actions, which are solely related to the unit itself, and interactive actions, which involve interactions with other units.
3.1.2 Multi-Task Imitation Learning
Since agents have to learn from broad range of tasks before they show good transfer to tasks, formulate the problem multi-task imitation learning. Formally, we assume access to dataset of passively logged expert trajectories D (T. = {(5,, ax)) > from some training tasks: 'T ~ p(T), where p(T) is the task distribution. After imitating the expert demonstrations through supervised learning from D, we then test the agents' performance on new tasks drawn from p(T). However, vanilla-supervised learning fails to produce good transfer in the multi-agent settings due to the mismatch between centralized pretraining and decentralized execution as well as the fact that the varying number of agents may lead completely new observation and action space that agents have never seen before, Then, we present our strategy to solve these issues.
3.2 Mask-Based Training Strategy
To handle the mismatch between centralized training and decentralized execution, we propose to randomly mask some parts of the units in the environment and collaboratively learn the execution policies of other unmasked units.
We utilize a standard causal transformer with only encoder layers as our model backbone, with the context length equal to L.. At the t'tth timestep, the input comprises the recent global states of N agents, expressed as (s" (s''-L+1 ... , resulting in total L N tokens, including all agents in the entire sequence, both controllable and uncontrollable. Then harness the capabilities of the attention matrix realize mask-based collaborative learning. The shape of these mask matrices is (LN LN), corresponding to L x N input tokens. During the training phase, meticulously craft the attention matrix my to exhibit causal structure along the timestep dimension, while maintaining non- causal configuration within each discrete timestep. This design is illustrated in the left part of Figure 3. Based on a predeternined mask ratio, we then generate the final attention matrix m2 for training (as depicted in the right part of Figure 3) through a process of random sampling, wherein specific elements are set to zero.
For evaluation, we can efficiently shift between centralized and decentralized execution by adjusting the attention mask matrix. For decentralized execution, modify the attention matrix ensure that each agent focuses solely the surrounding agents during the self-attention process. In contrast, for centralized execution, simply set the attention matrix to be equivalent to my.
3.3 Generalizable Action Representation
As agents involve interactive actions among other units in most multi-agent tasks (e.g., the healing allies and attacking enemies in SMAC), their action space grows with the number of units. To address the challenge posed by variable numbers of agents, we introduce the concept of Generalizable Action Representation (GAR), which similar to Yang et al. (2023). Figure 2 illustrates the process through which the final action at of specific agent is determined by GAR.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Random
Mask
Figure 3: Visualization of the attention matrix in MTS. Left: The attention matrix displays causal structure along the timestep dimension, complemented by a non-causal configuration within each discrete timestep. Right: The final attention matrix used training obtained by randomly masking elements of the left attention matrix.
Consider an interaction between the executor unit u, and the receiver unit uj. we define the embedding of this interaction as ,4 h hy, where hf and ht are the output embedding of u, and új from the transformer encoder respectively, where
and denotes the concatenation operation. For interactive actions, we compute the logits {FC((,,))) the fully connected layer FC has an output shape of one. In contrast, the logits for intrinsic actions are obtained using FC(h),, with the output shape of FC corresponding to the number of intrinsic actions. These logits, encompassing both interactive and intrinsic actions, are then combined and input into a softmax function determine the final action. Please refer to the pseudocode in the Appendix A.4 for the practical implementation.
3.4 Loss Function
With Mask-Base Mask-Based Training Strategy (MTS) and Generalizable Action Representation (GAR), our objective is to minimize multi-agent supervised learning loss. This objective is calculated using the cross-entropy loss between predicted actions and the ground truth actions. Our loss function for an action sequence is defined as:
11 ΣΣ CE(AS, L
LCE
where CB(a, p) is the cross-entropy between the ground truth action and the predicted action probability distribution. Here, M is the number of controllable agents, and L represents the context length. Note that for all agents in entire input T. x N, we only consider the loss for the M controllable agents.
4 Experiments
In this section, we design experiments to evaluate the following features of MaskMA. (1) Zero-shot Ability. We conduct experiments on SMAC using only 11 maps for training and up to 60 maps for testing, assessing the model's 's ability to generalize to unseen scenarios. In SMAC tasks, agents must adeptly execute a set of skills such as alternating fire, kiting, focus fire, and positioning to secure victory. These attributes make zero-shot transfer profoundly challenging. (2) Effectiveness MTS and GAR for different multi-agent tasks. We conduct ablation studies to figure out (3) Generalization of MaskMA downstream tasks. We evaluate the model's performance on various downstream tasks, such as varied policies collaboration, ally malfunction, and ad hoc team play. This helps us understand how the learned skills and strategies can be effectively adapted to different situations.
Setup. In SMAC (Samvelyan al., 2019), players control ally units in StarCraft using cooperative micro-tricks to defeat enemy units with built-in rules, and the state of unit s; includes unit type, position, health, shield, and so Our approach differs from existing methods that only consider grouped scenarios, such as Easy, Hard, and Super-Hard maps. Instead, we extend the multi-agent docisionn-aking tasks by combining different units with varying numbers. We include three races: Protoss (colossus, zealot, stalker), Terran (marauder, marine, and medivac), and Zerg (baneling, zergling, and hydralisk). Note that since StarCraft II does not allow units from different races to be on the same team, have designed experiments within this constraint. Firstly, collect expert trajectories offline datasets from
5

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1:
Win rate on training maps. Offline datasets consist of 10k or 50K expert trajectories per map collected by specific expert policies. With the mask-based training strategy, MaskMA consistently exhibits high performance in both Centralized Execution and Decentralized Execution. Its generalizable action representation enables seamless adaptation and convergence on maps with diverse characteristies. In contrast, MADT struggles with different action spaces, achieving win rate of only 51.78% even after extensive training.
Centralized Execution
Decentralized Execution
Map Name
Episode
Return Distribution
MaskMA (Ours)
MADT
MaskMA (Ours)
3$_ vs__2
50k
19.40 1 1.89
85.94 3.49
73.44 1 3.49
82.81 = 7.81
3s5z
10k
18.83 + 2.48
98.44 1.56
15.62 = 6.99
99.22 = 1.35
Ic3s5z
10k
19.51 + 1,40
94,53 + 4.06
54.69 + 8.41
95.31 = 1.56
2s57__s_3s66
10k
19.69 +1.27
85.94 6.44
14.84 = 9.97
85.16 5.58
5m_ vs_ 6m
10k
18.37 + 3.69
86.72 + 1.35
85,94 = 5.18
84.38 = 4.94
8m_ vs_ 9m
10k
19.12 + 2.57
88.28 6.00
87.50 = 2.21
86.72 = 4.06
MMM2
50k
18.68 + 3.42
92.97 + 2.59
62.50 11.69
86.72 4.62
2c_ ss_vs__e 64zg
10k
19.87 + 0.48
99.22 1.35
34.38 - 9.11
92.97 = 2.59
corridor
10k
19.44 + 1.61
96.88 3.83
21.88 = 11.48
94.53 2.59
6h_vs_8z
10k
18.72 + 2.33
75.00 5.85
27.34 - 6.77
76.56 = 6.44
bane bane
10k
19.61 + 1.26
96.09 2.59
91.41 + 4.62
98.44 1.56
Average
19.20 + 2.04
90.91 3.56
51.78 7.27
89.35 3.92
the 11 training maps by utilizing the expert policies trained with strong RL method named ACE (Li et al., 2022). This yields 11 offline datasets, most of which contain 10k episodes with average return exceeding 18. Then, we employ different methods pretrain on the offline dataset and evaluate their zero-shot capabilities on 60 generated test maps. As shown in Table I, we run 32 test episodes to obtain the win rate and report the average win rate as well as the standard deviation across 4 seeds.
Baseline We take the MADT (Meng et al., 2023) as our baseline for comparison which utilizes a causal transformer to consider the history of local observation and action for agent. MADT transforms multi-agent pretraining data into single-agent pretraining data and trains each agent's s policy independently, therefore adopting decentralized training and decentralized execution setting. Specifically, the input to each agent's ' policy is its own observation. Such an independent learning pipeline leads to an increase in computational complexity of O (N²) w.z. agent numbers N. Our experiments of MaskMA and baseline MADT utilize the same hyperparameters which are detailed in the Appendix.
4.1 Performance on Training Maps
Performance on the training domain is the most basic ability of one model, thus we first assess MaskMA and the baseline method on datasets including 11 training maps. As shown in Table 1, MaskMA achieves 90.91% % average win rate in 11 maps for Centralized Execution setting, while MADT can not be applied in this setting. Moreover, MaskMA surpasses MADT by significant margin on Decentralized Execution setting (89.35% v.s. 51.78%). In some challenging maps like 3s5z vs 3s6z, the advantage of our MaskMA is even much larger (85.16% v.s. 14.84%). One key observation is that MaskMA consistently performs well in both Centralized Training & Centralized Execution (CTCE) and Centralized Training & Decentralized Execution (CTDE) settings, highlighting its flexibility and adaptability in various execution paradigms. Figure 5a represents the curve of the average training win rate of MaskMA and the baseline method in 11 training maps on the Decentralized Execution setting. MaskMA significantly outperforms the baseline and achieves more than 80% win rate in most maps within 0.5M training steps, showing the robustness and efficiency of MaskMA.
4.2 MaskMA as Excellent Zero-shot Learners
Zero-shot generalization ability is the core motivation of our design, so we also evaluate the model's ability to generalize to extensive unseen scenarios without any retraining. Specifically, we evaluate our MaskMA and MADT on the 60 unseen testing maps. Table 2 shows that MaskMA outperforms MADT in zero-shot scenarios by large margin in

---

Published in Transactions on Machine Learning Research (09/2024)
Decentralized Execution setting (77.75% 43.72%), successfully transferring knowledge tasks without requiring any additional fine-tuning. In Centralized Execution setting, the performance of our MaskMA is even better with 79.71% win rate. These indicate that MaskMA's mask-based training strategy and generalizable action representation effectively address the challenges partial observation, varying agent numbers, and different action spaces in multi-agent environments.
Furthermore, MaskMA consistently performs well across varying levels of environment complexity, as demonstrated by the win rates in different entity groups. In contrast, MADT's performance is obviously dropped in high-complexity environments (i.e., # Entity 20). This highlights the ability of MaskMA to generalize and adapt to diverse scenarios, which is a key feature of a robust multi-agent decision, making it a versatile and reliable choice for multi-agent tasks.
Table 2: Zero-shot win rate on 60 unseen test maps after training only on 11 training maps. "# Entity" denotes the number of entities present in each individual map, while Map" represents the number maps fulfilling entity condition. Our MaskMA outperforms MADT by large margin in the zero-shot setting.
Centralized Execution
Decentralized Execution
# Entity
# Map
MaskMA(Ours)
MADT
MaskMA(Ours)
< 10
23
76.26 + 3.30
43.55 + 3.94
74.38 + 3.57
10 ~ 20
22
83.81 + 2.85
46.77 + 3.67
80.08 = 2.98
> 20
15
79.01 + 5.02
39.53 = 3.61
79.48 = 3.84
All
60
79.71 =: 3.56
43.72 +: 3.76
77.75 +: 3.42
4.3 Performance on Downstream Tasks
Evaluating MaskMA's generalization ability to downstream tasks can help us understand whether the learned skills and strategies could be effectively adapted to different situations. In this section, we provide various downstream tasks to assess MaskMA's robust generalization, including varied policies of collaboration, ally malfunction, and ad hoc team play.
Varied Policies Collaboration. In this task, some agents controlled by our trained policy (i.e., controllable agents), while others are controlled by an external baseline policy (i.e., uncontrollable agents). This task requires our trained policy to have excellent collaborative ability to cooperate with external policies. We conducted simulations in the 8m_vs_9m map, where our team controlled 8 marines to defeat 9 enemy marines. The uncontrollable agents are controlled by an external baseline policy with an average win rate of 41.41%. As shown in Table 3, MaskMA exhibits seamless collaboration with uncontrollable agents under different scenarios. MaskMA dynamically adapts to the strategies of other players and effectively coordinates actions, Specially, with only 50% controllable agents, the win rate has been improved significantly by nearly times (79.69% v.s. 41.4 11).1%)
Win rate on varied policies collaboration task with 8m_vs 9m_9 map. The uncontrollable agents controlled by policy with 41.41% % average win rate (i.e., the win rate with controllable agents). MaskMA demonstrates excellent collaborative performance in diverse scenarios with varying proportion of uncontrollable agents.
Table 3:
Proportion of Our Controllable Agents
0%
25%
50%
75%
100%
Win Rate
41.41 1 6.00
62.50 1:: 7.33 79.69 1 5.18
89.84 1: 2.59 86.72 1: 4.06
Ally Malfunction. In this task, one ally marine may become disabled or die due to external factors during execution. The time of ally marine malfunction is a changeable parameter. MaskMA is requested to handle such situations gracefully by redistributing tasks among the remaining agents while maintaining overall performance. Simulations are conducted in the 8m_vs_ 9m map, where we control 8 marines against enemy marines. The average win rate without ally malfunction 86.72%. As given in Table 4, MaskMA exhibits robustness and adaptability in the face of unexpected ally malfunction.
7

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Win rate on ally malfunction task with 8m_vs. 'm map. "Marine Malfunction Time" indicates the time of marine malfunction during episode. For instance, 0.2 means that one ally marine begins to exhibit stationary behavior at 1/5th of the episode. "None" signifies the original 8m_vs _0_ 9_m configuration without any marine malfunctions.
Ally Marine Malfunction Time
None
0.4
Rate
86.72 + 4.06 1.56 1.56
37.5 +: 6.99
71.09 + 6.77 86.72 2.59
Ad Hoe Team Play. In this task, agents need to quickly form a team with one new ally agent during the execution of the task. The challenge lies in the ability of the model to incorporate this new agent into the team and allow them to contribute effectively without disrupting the ongoing strategy. Simulations are conducted in the 7m_ 9'm map (i.e., our 7 marines to defeat 9 enemy marines), and the average win rate without extra ally marine is 0%. As shown in Table 5, MaskMA demonstrates excellent performance in ad hoc team play scenarios, adjusting its strategy to accommodate new agents and ensuring a cohesive team performance. We provide visualizations of MaskMA's behavior on ad hoc team play in Figure 4. More results are presented in the Appendix.
Table 5: Win on ad hoc team play task with 7m_vs_9pp0 ally marine during an episode. For example, 0.2 represents adding one ally marine at 1/5th of the episode. "None" signifies the original _s_ setup without any extra ally marine.
vs__m 9m map. "Marine Inclusion Time" indicates the time of adding an extra
vs_
Ally Marine Inclusion Time
None
0.2
0.4
0.6
0.8
Win Rate
0 010 0
80.47
7.12
78.12
2.21
50.00 + 8.84
10.94 6.81
(b)
(d)
Figure 4: Visualization of Ad Hoc Team Play on 7m vs 9m with Marine Inclusion Time = 0.8. This experiment demonstrates that when new Marines are added near the end of an episode, MaskMA still can quickly incorporate them into the team and enable them to contribute effectively. (a) Initial distribution of agents' positions. (b) Prior to the addition of the new Marine, our team left with only three severely wounded agents, on the brink of defeat. (c) The new agent (indicated by the red arrow) joins our team and immediately engages the enemy. (d) With the assistance of the newly added agent, our team successfully defcats the enemy.
4.4 Ablation Study
We perform extensive ablation studies to verify the effectiveness of proposed components and hyper-paramene choices.
Generalizable Action Representation (GAR). We compare GAR module to an alternative action representation method, i.e., aligning the maximum action length with a specific action mask for each task. As shown in Table 6, removing GAR leads to significant performance degradation (#2 v.s. #4) on the testing maps, emphasizing its importance in improving the model's generalization capabilities.
Training Strategy (MTS). The ablation results shown in Table 6. The alternative method of MTS is centralized training without masking, which naturally excels in the Centralized Execution setting (#3). However, the model without MTS struggles to generalize to Decentralized Execution setting, exhibiting significant performance degradation (#3 v.s. #4). The MTS employed in MaskMA, while posing a challenging training task, helps the model learn permanent representations stable representations across different scenarios and observation range. Intuitively, the random mask ratio aligns with the inference process, where the number of enemies and allies gradually increases in an agent's local observation due cooperative micro-operations, egg., positioning, kiting, and focusing fire.
8

---

Published in Transactions on Machine Learning Research (09/2024)
80
80
80
Win Rate %
Win Rate
Win Rate
40
40
Timestep
20
MaskMA
20
Timestep 10
20
MADT
Timestep 30
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2
8
11
Training Iterations (M)
Training Iterations (M)
Map Numbers
(a)
(b)
(c)
Figure 5: (a) Comparison of learning curve with the win rate on training maps in Decentralized Execution setting. MaskMA consistently outperforms MADT on average win rate. (b) Ablation on timestep with win rate on training maps in the Decentralized Execution setting. MaskMA performs better with longer timestep. (c) Ablation on the number of training maps with the win rate on unseen maps in the Decentralized Exccution setting. With the increasing of training maps (especially from to 8), the model's performance various unseen maps improves, indicating better generalization ability.
Table 6: Ablation over mask-based training strategy (MTS) and generalizable action representation (GAR). The results are the performance on training maps. baseline a naive transformer. Each row adds new component to baseline, showcasing how each modification affects the performance.
Module
Setting
MTS
GAR
Cent. Exe.
Decent. Exe.
44.67 1:: 3.35
8.03 + 1.44
V
39.49 + 3.05
39.91 +: 3,97
3
/
91.26 + 4.21
41.55 + 4.38
/
90.91 + 3.56
89.35 + 3.92
Mask Ratio in MTS. We explore different types of masks for training, including of fixed mask ratios, local mask, and random mask ratios chosen from [0, 1] for units each time step. We provide mask ratio analysis in Table 7. The results show that as the masking ratio increases, the performance the training maps increases in the Decentralized Execution setting while decreasing in the Centralized Execution setting. This suggests that an appropriate masking ratio helps strike a balance between learning useful representations and maintaining adaptability to dynamic scenarios in the agent's local observation. In conclusion, a random mask ratio with the range [0, 1] is simple yet effective way to balance the two settings, which could combine the advantages of various ratio masks. This approach allows MaskMA to demonstrate strong performance in both centralized and decentralized settings while maintaining the adaptability and generalization necessary for complex multi-agent tasks.
Timestep Length. We conduct experiments with different timestep length to study the access to previous states. As shown in Figure 5b, MaskMA's performance is better when using a longer timestep length. One hypothesis is that the POMDP property of the SMAC environment necessitates that policies in SMAC take into account sufficient historical information to make informed decisions. Considering the balance between performance and efficiency, we select timestep K 10 as default typer-parameter. This choice allows MaskMA to leverage enough historical information make well-informed decisions while maintaining reasonable level of computational complexity.
Number of Training Maps. Figure 5c demonstrates the influences of the number of training maps. As the number of training maps increases, the win rate on the testing maps also improves, indicating that the model is better equipped to tackle new situations. marked uptick in win rate is observed when the map count rises from 8, underlining the value of training the model across varied settings. This trend in MaskMA offers exciting prospects for multi-agent decision-makinn It implies that by augmenting the count of training maps or integrating richer, more intricate training scenarios, the model can bolster its adaptability and generalization skills.

---

Published in Transactions on Machine Learning Research (09/2024)
Table 7: Ablation study mask ratio for training. "Local Mask" represents using local visibility the agent as mask.
Mask Ratio
Setting
Local Mask
0.2
0.5
0.8
Random [0. 1]
Centralized Execution
55.97 + 4.67
91.26 + 4.21
89.70 = 3.81
88.21 + 3.78
82.81 + 4.83
90.91 + 3.56
Decentralized Execution
83.59 = 8.08
41.55
4.38
58.03
5.70
71.52
4.23
82.03 = 5,01
89.35 = 3.92
Training Cost and Parameter Count. MaskMA processes the inputs of all agents concurrently, achieving a notable degree of parallelism superior to MADT, which transforms multi-agent training data into single-agent data. Conse- quently, MaskMA considerably more time-efficient than MADT when trained over identical epochs. Specifically, training 2M iterations on one A100 GPU with the same model architecture and the same 190K training data, MaskMA needs only 31 hours, whereas MADT requires 70 hours, Also, the parameter counts for both models are nearly identical since they share the transformer architecture and the only difference lies in the final fully connected (FC) layer for action output.
5 Discussion
We have introduced MaskMA, mask-based collaborative learning framework for multi-agent decision-making to tackle the challenges of zero-shot generalization in the multi-agent rocision-maaing field. MaskMA consists of transformer architecture, a mask-based training strategy, and a generalizable action representation. Extensive experiments on SMAC dataset demonstrate the effectiveness of our MaskMA in terms of zero-shot performance and adaptability to various downstream tasks, such as varied policies collaboration, ally malfunction, and hoc team play.
Limitations & Future Work Our current study primarily relies on expert datasets. Exploring how to utilize low- quality demonstration datasets to develop a generalist agent with robust zero-shot capabilities presents a promising research direction. Furthermore, extending the maskMA approach to create a single generalist agent capable of generalizing across different types of environments, such entirely different games, is also promising avenue for future research.
Acknowledgment
This work was supported by the JC STEM Lab of AI for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust.
References
Susanne C Brenner. The mathematical theory of finite element methods. Springer, 2008.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Micah Carroll, Orr Paradise, Jessy Lin, Raluca Georgeseu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, et al. Uni [mask]: Unified inference in sequential decision problems. Advances neural information processing systems, 35:35365-35378, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
DI engine Contributors. DI-engine: OpenDILab decision intelligence engine. http://ghhhal.commoppppllla//ggp 2021.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey. Artificial Intelligence Review, pp. 1-49, 2022.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rl via policy decoupling with transformers. International Conference on Learning Representations, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-992999999 9915.991... PMLR, 2022.
Chuming Li, Jie Liu, Yinmin Zhang, Yuhong Wei, Yazhe Niu, Yaodong Yang, Yu Liu, and Wanli Ouyang. Ace: Cooperative multi-agent q q-learning with bidirectional action-dependency. In Proceedings of the AAAI conference artificial intelligence, 2022.
Fangchen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. Advances in Neural Information Processing Systems, 12122-2206611 2022.
Linghui Meng, Muning Wen, Chenyang Le. Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, and Bo Xu. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-248 2023. ISSN 2731-538X. doi: 10.1007/s 16300/02111---0222-88-- URI. https//wwwwmiirrssarcch. net/cn/article/doo//00 10. 10100000011/330000 1383-7.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized pomdps, 2015.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-800 8037, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204006125, 2022.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226-237. 2022a.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226-237, 2022b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Weixun Wang. Tiampei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. In International Conference on Learning Representations, 2020.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning sequence modeling problem. Advances in Neural Information Processing Systems, 35: 16509-16521, 2022a.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang. Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. Advances in Neural Information Processing Systems, 35: 16509-16521, 2022b.
Jiachen Yang, Tarik Dzanic, Brenden Petersen, Jun Kudo, Ketan Mittal, Vladimir Tomov, Jean-Sylvain Camier, Tuo Zhao, Hongyuan Zha, Tzanio Kolev, et al. Reinforcement learning for adaptive mesh refinement. In International Conference on Artificial Intelligence and Statistics, pp. 5997-6014, PMLR, 2023.
Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2023.
12

---

Published in Transactions on Machine Learning Research (09/2024)
In this Supplementary Material, provide more elaboration the implementation details and experiment results. Specifically, we present the implementation details of the model training in Section A and additional results and visualization in Section B.
A Additional Implementation Details
In this section, provide detailed description of the required environment, hyperparameters.
A.1 Environment.
We use the following software versions:
CentOS 7.9, Python 3.8.5, Pytorch 2.0.0 (Paszke et al., 2019)
StarCraft II 4.10 (Samvelyan al., 2019), DI-engine 0.2.0 (engine Contributors, 2021)
We conduct all experiments with single A100 GPU.
A.2 Hyperparameters
As shown Table 8, our experiments of MaskMA and baseline MADT (Meng et al., 2023) utilize the same hyperpa- rameters.
Table 8: Hyperparameters of MaskMA and MADT. It should be noted that both models utilize the exact same set of hyperparameters.
Hyperparameter
Value
Optimizer
RMSProp
Learning rate
lc-4
Training
Batch size
256
Weight decay
le-5
Number of blocks
Hidden dim
128
Architecture
Number of heads
Timestep length
10
A.3 Differences between the training and testing maps
We list some differences between the training and testing maps in Table 9.
Difference
Testing Maps
Most Similar Training Maps
Number of agents
10m_ys lmm
$m_vs_6m
Different race matchups
Protoss Terran
Protoss Protoss, Terran vs. Terran
Reversed ally and enemy
64zg_vs. vs__e__________ 2c
2c, vs_64zg
Table 9: Comparison of testing maps and training maps
A.4 Pseudocode
The Pytorch-style implementation of the MaskMA is shown below:
13

---

Published in Transactions on Machine Learning Research (09/2024)
a: controllable, ue: uncontrollable
Input: state_enaa certt_enbedding for c_entity [s_1, s_M] and uc_entity [s_M+1, $_N] Initial: func: GenerateMask
Enhance state eeteeednmeddngg with information of timestep and controllable
state s embeddiin_e state_ caate__ebedding_S timestep eapppdimmbddn + controllable n
state embedding state caste__mbedding..o
eeneedigpembedding + uncontrollable enbeedinn
Mask-based Training Strategy
Generate random mask
mask GenerateMask (mask_ratio, state_embeddinngcc state_ caate__mbedddin_ugg
Apply masks and process through the transformer
hidden state_c, hidden state_uc transformer (state_ embeddingg_;
state embedding uc, mask)
actions GAR (hiddddenn_stee state__ hidden state_uc) uc)
Input:
[B, T, M, D], hidden_state_us [B, N-M, D] Initial: FC layers: fo fo_interactive, fc fo_intriisic
def GAR (biiddensettte_c,, state_e, sidden_state.de Interactive actions
hidden state torch.cat (hidden state_e, hidden state state_uc], dim=-2)
Concatenate
repeat embeddings hidden state for prediction
hidden state_cc___ccce c_rpeat hidden_staad stt.sstttttsusteeeeeeeeeee unsuuueeee (3) repeat (1, 1, 1, N, 1)
hidden_ starenstttteeeeeeeeeeeeeeeeeeeee _epeat hidden_ state. unsqueeze (2) repeat (1, 1, M, 1, 1)
interactive actaacctteeeeeeee iesiion_mbeedding torch. ([hidden_stee Baciiie___tttt________e c_repatt,
Salden_state_reeee
dim=-1)
Process result through letack_actionn network
interactive logits fc interacttee (interactive action aetticneeeeeeteieeeeeee embedding)
Intrinsic actions ierinniccactiinn in = hidden hidden_state_ state_c
Compute intrinsic logits using action queries
Antrinsic (egiitriinsiccinttnnnicccc actiion action embedding)
Final actions
Concatenate logits to form action predictions action preds torch. ([interactive intrinsic logits), dim=-1) Determine final actions by selecting the maximum valid logits actions action aettio_preds_.._____ee m-x(-1) indices return actions
A.5
State units
In the original StarCraft Multi-Agent Challenge (SMAC) setting, the length of the observation feature fluctuates in accordance with the number of agents. To enhance generalization, MaskMA directly utilizes each unit's state input the transformer architecture. As depicted in Table 10, within the SMAC context, each unit's state comprises 42 elements, constructed from nine distinct sections. Specifically, the unit type section, with length of 10, represents the nine unit types along with additional reserved type.
B Additional Results and Analysis
B.1 Win rate of all maps
As shown in Table present the win rate of MaskMA and MADT on 11 training maps and 60 testing maps.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Table 10: The composition of state.
State name
dim
ally enemy
unit type
10
pos.x and y health shield cooldown
last action
7
path height
9
9
B.2 Comparition pretraining dataset size
The scale of the pretraining dataset significantly impacts the eventual performance. the multi-agent StarCraft II environment, SMAC, we investigated the optimal size for the pretraining dataset. We take the 3s 3s_vs_5z map an example and solely the pretraining dataset of this map to train MaskMA, and then test it the same map. As illustrated in Figure 6, a dataset encompassing 1 episodes was found insufficient, leading to progressive decline in win rates. In contrast, dataset comprising 50k episodes demonstrated exceptional performance. Specifically, for the 3s_vs_5 5z and MMM2 maps, a pretraining dataset containing 50k episodes proved appropriate. For the remaining nine maps, a dataset consisting of 10k episodes was found to be suitable.
100
80
Win Rate %
mWy
60
40
20
Episode Number 50k
Episode Number 10k
0.0
0.5
1.0
1.5 2.0
Training Iterations (M)
Figure 6: Performance comparison across different sizes of the pretraining dataset on the 38 vs. 52 Map.
B.3 Visualization
In this section, provide visualizations of MaskMA's behaviors additional two downstream tasks: varied policies collaboration, and teammate malfunction. Figure 7, 8 evaluate the strong generalization of MaskMA. Additionally, we offer replay videos for more comprehensive understanding of MaskMA's behavior and strategies.
15

---

Published in Transactions on Machine Learning Research (09/2024)
CHEAT ontance KIIlwBBy 78548
CHEAT aomnn Kgl.mi Tag 28P145
EHEAT Kataniyy (
kithniyy lg Pesess
CHEAT
a ban Kilwwiy Tag 048577
CHEAT celioon Kiu mayy og 262 45" Cre quiaooonnn Kabiiity my 1800791
CHEAT
TuuuiByTap (33009
KilhiByTas
(a)
(b)
Figure 7: Varied Policies Collaboration 8m_vs un__s__ 9m with 4 Agents with different policies. The agents within the green box are controlled by other policies (replaced with a network trained with a 41% win rate), while the agents within the red circle are controlled by MaskMA. (a): Initial distribution of agents' positions. (b): MaskMA dynamically adapts to the strategies of players different policies and effectively coordinates actions, resulting in a victorious outcome.
KILKIB
KaTLAaa 1 17885
wiuhaai 4330116
KilL 1048560
(b)
Figure 8: Teammate Malfunction 8m_vs_9m with Marine Malfunction Time = 0.6. The agent within the red circle suddenly malfunctions in the middle episode, remaining stationary and taking no actions. (a): The agent within circle starts malfunctioning. (b): Despite the malfunctioning teammate, other agents continue to collaborate effectively and eventually succeed in climinating the enemy.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Table 11: Win rate MaskMA and MADT on training maps and testing maps.
Ours
MADT
Map
Centralized
Execution
Deventralzzed Exccution
Decentralized Execution
85.94
4-7.11
73.44 4:3.9
3s5z
98.44 11.56
99.22 11.35
15.62 +6,99
lc3s5z
94.53 =4.06
95.31 +1.56
54,69 =8.41
3s52_ya, 3s62
85.94 =6.44
85.16 +5.58
14.84 =9.97
5mL_ vs_ 6m
86.72 11.35
84.38 1:4.94
85.94 15.18
8m 9m
88.28
86.72 1.406
87.50 12.21
MMM2
92.97 +2.59
86.72 +-4.62
62.50 11.69
2e_vs, ys_ 642g
99.22 11335
92.97 +:2.9
34,38 11
corridor
96.88 +3.83
94.53 +2.59
21.88 111448
vi_vs__z
75.00 =5.85
76.56 +6.44
27.34 =6.77
bane bane
96.09 12.59
98.44 1:.56
1:462
2s_vs__
100.00=0.00
100.00=0.00
70.31 100.0
3z__ vs_3s
71.88 +3.83
53.91 19.47
0.00 1:0.0
lc2s
44.53 1.5.58
41.41 1.6.9
2.34 1.2.59
4z, vs 3s
99.22 1.35
97.66 1:335
4.69 4-2.71
99.22
100.00=0..00
28,91 +6.00
Ic3s
54.69 33.49
48.44 +:469
17.19 #4.69
5m_ys_4m
100.00=0.00
100.00=0.00
100.00=:0.00
5m
100.00-0.00
100.00
98.44 1.55
2x32
10,94 1.518
8.59 1.2.59
0.00 1:0.00
1s42_vs Ima4m
91.41 #4.62
88,59 #3.41
35.16
362r_v__2$$3
9.38 15.85
11.72 +2.59
0.00 1:.00
263,ys. vs_ 1na2m2me
92.97 14.06
82.03 144.04
35.16 19.73
2632, vs_ 1ma3m Imm
90.62 43.12
93.75 +3383
37.50 77.97
lc3slz_ vs_ Ima4m
100.0000.00
99.22 +1.35
1.56
1s4z_vs. 2ma3m
83.59 #7.12
77.66 +3.41
73.44 66.44
Ic3slz_vs_0 2533
100.00=0.00
100.00=0.00
78.12 =5.85
2s32_vs_ 1h42g
96.88 13.12
96.09 13.41
71.88 17.16
3x2a_ys_ 2h_3g
95.31 15.18
93.75 1.442
75.00 13388
1c4x_yx. 2x3z.
100.0000.00
100.00
71.88
2ma3m_ me
46.09 +12.57
39.06 +99.44
0.00 +0.00
2537_v5 vs_ lb_lbzg
100.00=0.00
96.88 +2221
67.19 #4.06
1s4z_vs_5z
3.12 =2.21
5.47 +4.06
+0.00
lels3z_ Ielk
64.06 #7.16
76.56 1:5.3
35.94 26.44
97.66 1.2.99
99,22 11135
47.66
93.75 +.8.3
87,50 4-2.21
58.59 1:35
Sr_vs_6h
93.75 +2221
98.44 +1.56
38.28 +4.06
2s5z
17.97 4.06
7.03 +3.41
0.78 +1.35
1c3s12_vs_ vs__i_s___llihgggg
100.00=0.00
100.00=0.00
$7.81 =0.00
1c3sl7_vv 1h97g
100.0000.00
100.00=0.00
60.94 +0.00
2o1s27_vs_lb92g
100.0000.00
100.00=0.00
49.22 =0.00
3ma6m1me. vs 3h2zg
100.00=0.00
100.00=0.00
23.44 =0.00
3s2e vs_ vs__hhh
97.66 1.33
85,94 14.69
52.34 22.59
Smadmime 1s_3hbli
100.00-0.000
100.0010.00
12.71
4ma5m1me _5_gg_
100.0000.00
100.00
85.94 48.12
1s5z_vs_4mmnn vs_4mm6m
39.06 +10.00
16.41 +3.41
4.69 +4.69
ima7m_vs. _o___s___38 2532
98.44 11.56
100.00=0.00
35.16 +13.07
2ma9m_ 3s22
89.06 =2.71
80.47 +6.0
51.56 =9.50
5s2a.vs 2ma8m
48.44 19.24
29. 29.69 111.16
8.59
2x52_vs 9m1mm
99.22 11.35
99.22 1.1.35
46.09 14.94
Icds_vs 23__0s_3
85.94 4.5.63
89.84 4-5.58
43.75 44.94
3872_ vs. 5mu2m3me
99.22 1.33
96.88 4:3.33
60.94 +2.59
Ic3s6z
58.59 17.12
61.72 +7777
14.84 =6.77
2c2s62_vs_ vc_16366
100.00=0.00
100.00=0.00
97.66
1c2s72
25.00 19.63
25.78 15.12
7.81 #3.49
1e4554_vs_ vs_ vs__mmmmmmmeeee
0.00
83.59 188.88
84.38 10.00
10s1z_vs, vs_ (aer_____h2hh/// 1h2h7rg
96.09 +4.06
96.88 4:2.21
68.75 +-1.6
10m, ys_ Imm
79.69 +6.44
78.91 4:7.45
0.00
1b10zg
85,16
89.06 47.16
60.16 12
2b10rg
92,19 +4.69
98.44 +1.56
30.47 +55.88
le vosss_333 vs_32ng
59.38 49,38
55,47 +:777
1.567
32zg. lc
92.97 =3.41
94.53 +33.41
53.91 =3.12
1b202g
74.22 10091
82.81 12.71
66.41 27.45
2b202g
100.00=0.00
97.66 1.33
71,09 14.94
36202g
96.09 13,41
88.28 14.62
85.94 14.69
16z vs. 6h242g
97.66 4.559
97.66 +-2.99
71.09 44.06
5b207g
99.22
100.00=0.00
32.03 +2.59
64zg_vs
55.47 +6.3
56.25 +2.21
48.44 88.41
1c8z_vs_6e v5__222g
89.06 17.16
85.94 15.18
2.34 +2.59
vs_ 1e8e_vs_
vs_2b6
61.72 17.77
63.62 15.85
0.00 10.00
1c8r vs, vs_5b2
6.25 12.21
4.69 1.3.49
0.78 11.35
17


Reviews:
[{"review_id": "viLfGXq52i", "reviewer": "Reviewer_rbcy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It is good to see that two simple methods lead to significant improvement in results over baseline\n2. Thorough and strong empirical results back up all the claims.\n3. This paper pushes forward the study of transformer-based policies for decision-making.\n\nWeaknesses:\n1. The \"Generalizable Action Representation\" that is proposed to help support a varying action space size is claimed as a new contribution, but it has appeared in past work where the action space changes, e.g. [1]\n2. In the formulation Section 3.1.1, the global state is defined by the set of all agent states. This formulation excludes environments that have agent-independent global state components.\n2. The model architecture is not completely described, so reproducing the model may be difficult.\n3. The paper acknowledges the limitation of the assumption that expert data is available, but does not discuss how this can be overcome.\n\n\n[1] Yang et al. \"Reinforcement learning for adaptive mesh refinement.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes two main contributions for cooperative multi-agent decision-making in the context of transformer-based policies: 1) random masking to improve generalization when the number of agents changes from train to test; 2) handling variable number of agents by using a fully-connected layer with output dimension 1 to compute the logits of interaction actions. Empirical results and ablations in the SMAC benchmark show that these two methods lead to improvement in zero-shot generalization over a previous transformer-based method, are robust to environment perturbations (variable number of policy-controlled agents, ally malfunction, ad-hoc team play).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* [critical] Revise statements that imply the novelty of the Generalizable Action Representation, and include references to suitable prior work that address varying size of action space.\n* [minor] In the Introduction first paragraph, \"different maps\" is mentioned without any prior context, it should be rephrased. Up to that point, readers do not know that this paper is concerned with the SMAC environment only.\n* [minor] In the Figure 1 caption, MADT is mentioned without any reference. Up to that point, readers do not know what is MADT.\n* [minor] In the Introduction page 2, the paper writes \"MaskMA is the first approach that achieves strong zero-shot capability for multi-agent decision-making\". It is not clear what \"strong\" means, make it more precise.\n* [minor] In Section 4.4 \"Mask-based Training Strategy\", there is a mention of \"learn permanent representations\". Rephrase this to be more precise about what \"permanent\" means and provide evidence for this \"permanence\", or rephrase the text.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8dCshzrPWt", "reviewer": "Reviewer_HV2H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Good research problem and motivation, i.e., the zero-shot ability of MARL;\n2. The framework of the algorithm is clear;\n\nWeaknesses:\nThe experiments proposed in the paper are constructive. However although the experimental scenarios of StarCraft II look rich, the similarity between StarCraft II scenes is very large which weakens the final conclusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A Mask-Based collaborative learning framework for Multi-Agent decision making is proposed. A self-supervised learning scheme is discussed with the purpose to establish the generalization ability. The authors verify the ability of this aspect Through some StarCraft II experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More complicated scene experiments need to be discussed, especially the authors should prove or measure the \"zero-shot\" distance between scenarios.\nBased on these results, the effectiveness of the method proposed by the method can be verified much more convincing.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KnlZj8gxrX", "reviewer": "Reviewer_YL12", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem considered in the paper is important. Connecting the sequence modeling literature with multi-agent decision-making is an interesting avenue of research.\n- The empirical results presented in the paper are quite strong. The proposed improvements do seem to matter a lot.\n- The paper offers valuable ablation studies and analysis of interesting tasks, such as combining neural agents with scripted agents, introducing new agents during the episode, etc. \n\nWeaknesses:\n- There are some issues with the presentation:\n    - GAR is poorly described. This is a central part of the paper but the idea is only explained in two paragraphs and a part of a figure. After reading the paper carefully I don't have a precise understanding of how it works. Why are there as many actions as there are states in the current timestep? I would appreciate formal equations or a pseudocode that explains GAR.\n    - The notion of interactive and intrinsic actions is not very well described. Can we provide a general formulation in the MDP framework about what “interactive” means? Having an explicit list of interactive/intrinsic actions in SMAC would be useful. I looked briefly at other papers and I haven't found definitions there. \n    - Some of the citations look broken, e.g., the [Carroll et al.] one does not include the year.\n- Also, I don't think the paper explains why GAR works so well. In particular, the transformer is already able to combine the embeddings across different agents through attention. Would it be possible to implement GAR using a single (or a few) attention layers? If not, then what part is specifically impossible for attention? If yes, why GAR helps?\n- Having more baselines that are standard in SMAC would be nice. For example, the MADT paper also includes BC, BCQ-MA, CQL-MA, and ICQ-MA. Evaluating some of the papers mentioned in the related work would also be beneficial. This is more of a \"nice-to-have\" than a major weakness. I would be also okay with mentioning this point in the limitations section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method for multi-agent sequential decision problems based on imitation learning using the transformer architecture. In particular, the authors use the Multi-Agent Decision Transformer framework proposed in prior work and modify it to better capture the specifics of multi-agent decision problems. In particular, they introduce mask-based training, which allows for centralized training and decentralized execution, and generalizable action representations, which allow the model to better handle interactive actions. The authors test the proposed approach on the StarCraft Multi-Agent Challenge with various scenarios and variations, showing that MaskMA outperforms MADT. They also run several ablation studies to identify where the improvements come from.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please address the issues with the presentation mentioned above.\n- Could you provide a hypothesis for why GAR works so well? In particular, I'm curious to understand what it can do that MADT (or a transformer in general) can't.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initial concerns were all addressed during the rebuttal phase.
Decision: Accept (poster)