MA$^2$E: Addressing Partial Observability in Multi-Agent Reinforcement Learning with Masked Auto-Encoder
Authors: Sehyeok Kang
Yongsik Lee
Gahee Kim
Song Chong
Se-Young Yun
Venue: ICLR 2025 Poster (2025)
Keywords: multi-agent reinforcement learning
partial observability
masked autoencoder
Primary area: reinforcement learning

Abstract: Centralized Training and Decentralized Execution (CTDE) is a widely adopted paradigm to solve cooperative multi-agent reinforcement learning (MARL) problems. Despite the successes achieved with CTDE, partial observability still limits cooperation among agents. While previous studies have attempted to overcome this challenge through communication, direct information exchanges could be restricted and introduce additional constraints. Alternatively, if an agent can infer the global information solely from local observations, it can obtain a global view without the need for communication. To this end, we propose the Multi-Agent Masked Auto-Encoder (MA$^2$E), which utilizes the masked auto-encoder architecture to infer the information of other agents from partial observations. By employing masking to learn to reconstruct global information, MA$^2$E serves as an inference module for individual agents within the CTDE framework. MA$^2$E can be easily integrated into existing MARL algorithms and has been experimentally proven to be effective across a wide range of environments and algorithms.

Published as a conference paper at ICLR 2025
MA E: ADDRESSING PARTIAL OBSERVABILITY IN MULTI-AGENT REINFORCEMENT LEARNING WITH MASKED AUTOENCODER
Sebyeok Kang
Yongsik Lee
Gahee Kim'
Song Chong
Se-Young Yun't
KAIST All
(kangsehyeok0322) dldydt1r99 gaheekim, songchong, yunseyoung @kaist.ac.o kr.
ABSTRACT
Centralized Training and Decentralized Execution (CTDE) is a widely adopted paradigm to solve cooperative multi-agent reinforcement learning (MARL) prob- lems. Despite the successes achieved with CTDE, partial observability still limits cooperation among agents. While previous studies have attempted to overcome this challenge through communication. direct information exchanges could be restricted and introduce additional constraints. Alternatively, if an agent can infer the global information solely from local observations, it obtain global view without the need for communication. To this end, we propose the Multi-Agent Masked Auto-Encoder (MA?R), which utilizes the masked auto-encoder architecture infer the information of other agents from partial observations. By employing masking to learn to reconstruct global information, MA serves as an inference module for individual agents within the CTDE framework. MA?E can be easily integrated into existing MARL algorithms and has been experimentally proven to be effective across a wide range of environments and algorithms. The code is available httttps://ithubb com htt.mm/hheeeeeee/eee/e
INTRODUCTION
In cooperative multi-agent tasks, the environments are typically partially observable, where individual agents do not have complete access to the global information. For instance, consider motivating scenario where allied and enemy forces are engaged in combat as illustrated in Figure An ally agent 1 might perceive the current situation as favorable by observing only one enemy, while ally agents and 3 observe four and five enemies, respectively, and therefore assess the situation as unfavorable. This discrepancy makes it difficult for the agents to make cooperative decisions. Therefore, addressing partial observability is crucial challenge in the field of cooperative Multi-Agent Reinforcement Learning (MARL) (Nguyen et al. 2020 Canese al., 2021 Ning & Xie 2024).
One straightforward approach is fully centralized training and control of joint policy for all agents which simplifies the MARL problem into single-agent one. However, the centralized setting suffers from scalability and heavy computational costs as the number of agents increases due to the curse of dimensionality (Gronauer & Diepold 2022 Du & Ding. 202T: Canese et al.) 202T)
Centralized Training and Decentralized Execution (CTDE) paradigm provides a structured frame- work for mitigating aforementioned issues (Canese et al. 2021 Ning & Xie 2024: Du & Ding 2021: Gronauer & Diepold 2022). but still fails to fully resolve the problem of partial observability. Although the global information is used during training, each agent relies solely on its local observa- tions during execution. Such discrepancies between learning and execution can limit collaborative
decision-making (Shao et al.) 2022 Yuan et al. 2022 Guan et al. 2022).
Previous works have explored the strategies relieve drawback of decentralized execution. Relaxing CTDE with communication by allowing direct messaging among agents have been extensively studied, but it faces limitations real-world scenarios where inter-agent communication is not permitted or is
"qqual contribution. 1 Correpponding authors.

---

Published as a conference paper at ICLR 2025
constrained by various factors such limited bandwidth and noisy channels (Zhu al. 2022 Ning
& Xie (222). Without communication, shared structures such as common knowledge (Schroeder de Witt et al. 20199 or one-hot consensus (Xu et al. 2023) have been proposed to foster cooperation. However, such abstractly encoded representations may be lossy in complex environments.
Masked modeling, which learns reconstruct original data from masked input, has shown notable achievements in the language (Devlin et al. 2018 Liu 2019; Clark, 2020), the vision (Chen al. 2020 Bao et al. 2021 He et al.)
O Ayy
Enomy
Ossenation rango
ot
St
2022), and even in the single-agent agent reinforce-
of
ment learning (RI.) domains (Seo et al. 2023
Liu et al.) 2023a] Wu et al. 2023). Inspired by those successes, in this paper, we propose Multi-Agent Masked AutoEncoder (MA"E)
of
which enables agents to infer the global in- formation based solely on partial observations. Our key idea is to use a masked autoencoder (MAE) (He et al., 2022), which masks sub- set of the input and is trained to recover the missing (masked) data using the visible (un- masked) regions, to reconstruct full trajectories of all agents from local observations. While such masked modeling usually targets to solve the downstream tasks with complete data after training from incomplete input, our focus is to utilize MAE's capability to handle partially observable data to infer the entire information. MA?E, MAE performs masking on a per-agent basis and reconstructs the original global input, allowing the inference of global information from partial observations. This enhances the agent's 's ability to deduce the situations of other agents and aids in making appropriate decisions. Furthermore, by separately training MA E from agents' policies, M can be easily plugged into existing MARL algorithms as backbone networks and the learned MAE can be transferred to other backbone policies.
A
A motivating combat scenario. The solid circles and red triangles denote the ally agents and enemies, respectively. The dotted circles are obser- vation ranges of each ally agent. While the current global state 8, is identical, each agent perceives the situation differently due to the discrepancy among their partial observations of, o².
Figure 1:
We experimentally evaluate our approach on the Starcraft Multi-agent Challenge (SMAC) (Samvelyan al. 2019), SMACv2 (Ellis et al. 2023), and Google Research Football (GRF) (Kurach al.) (020) environments. The experimental results consistently demonstrate that MA?E achieves faster con- vergence and higher sample efficiency compared to fine-tuned QMIX (Hu et al.) 2021). which is the state-of-the-ar arrt MARL algorithm. Additionally, MA shows comparable or superior perfor- mance compared to the cases where full observations are provided or communication is employed, substantiating the ability of M to effectively infer full observations from partial observations. Furthermore, employing MA achieves performance gains for various value-based and policy-based MARL methods as backbone algorithm. Finally, to effectively integrate MA E into the backbone MARL algorithms, we propose an appropriate configuration for applying M across different hyperparameterrs and masking strategies in ablation studies.
2 RELATED WORK
Partial Observability in MARL: Partial observability is fundamental challenge in cooperative MARL. As a naive adoption of a fully centralized setting results in an intractable computational com- plexity (Canese et al., [ | Du & Ding 2021 Gronauer & Diepold 2022). alternative strategies have
been explored. Centralized Training and Decentralized Execution (TTE) is popular framework to
solve partial observability (Canese et al. 2021 Ning Xie 2024 Du Ding 2021 Gronauer Diepold 2022). In value-based CTDE algorithms. value decomposition which factorizes joint value function into individual ones is dominant (Sunchag et al. 2017 Rashid et al. 2018 Lowe et al. 2017]
Foerster et al. 2018; Yu et al. 2022). Another major branch is basec CTDE methods, where actor-critic approach is pudelyyadoptee
(Lowe et al. 2017 Foerster et al.) 2018 Yu et al. 2022). We
aim to alleviate the limitation of fully decentralized execution solely with local observations while following the CTDE paradigm. Subsequent to some pioneer works (Sukhbuatar et al.) 2016 Foerster
al. 2016), communication among agents has been extensively studied in cooperative MARL.
Yuaan
2

---

Published as a conference paper at ICLR 2025
Tp-T+2
Recontructed Trajectries
2
Decoder
Encoder
Maaking
Input
Trajectories
Figure 2: The architecture of MA?E, During centralized training. MA masks out l agents' trajecto- ries from all agents' trajectories 7 and learns reconstruct them. The positional encoding is applied considering both the time and the agent information. During decentralized execution, trajectories of other agents are masked except the trajectories of current agent.
et al. 2022, Hu et al. 2021 2024). Although communication can mitigate partial observability, it
may not be possible in target environments or constrained by diverse factors (Zhu et al.) 2022 Ning
& Xie, 2024). Li al. 2023) gradually shifts from communication tacit collaboration, but
requires communication during exploration. Our goal is to remove the dependence on communication. Another line of research is utilize shared structures dependent only local information during execution. MACKRL (Schroeder de Witt et al. (019) employs common knowledge among groups of agents with a hierarchical policy tree. COLA (Xu et al. 2023) infers the same one-hot consensus for all agents from different local observations. Compared to those limited abstractions, our method enables each agent recover the global trajectories of all agents from its local observations.
Masked Modeling in Reinforcement Learning: Masked modeling to reconstruct original data from masked ones emerges as a powerful technique diverse domains including vision (Chen et al.) 2020)
Bao et al. 2021 He et al. 2022) and NI.P (Devlin et al. [0188 Liu 2019 Clark] (020). Recently.
the idea has been extended the RL field. Masked World Model Seo et al.) (023) trained to
reconstruct pixels from masked convolutional features to learn latent dynamics model. MaskDP
(Liu et al. 2023a) and MTM (Wu al., 2023) apply masking to a portion input trajectories and learn reconstruct them. They show generalization abilities diverse tasks by manipulating masking patterns desired for the target task. Aforementioned methods aim to solve the single-agent RL tasks. In contrast, explore MARL problems with masked modeling. MA2CL (Song et al.) 2023) deploys masked modeling with contrastive loss. The policy and the reconstruction module are jointly trained to obtain encoder representation for the policy with enhanced collaboration. In contrast, M separately trains the policy and the reconstruction module, and its main focus is to recover global information rather than encoder representation. MaskMA (Liu al. (023b) treats MARL as sequence modeling problem and masking is used to predict next action, rather than reconstructing the inputs. contrast, our aim is recover global information using MAE for decentralized execution.
METHOD
Problem Formulation: We consider cooperative MARL task which can be described as a Dec-
POMDP (Oliehock et al., 2016). Dec-POMDP is defined as a tuple G = (S, U, 0, 7) where denotes the global environment state. At each time step, each agent i {11,,, {1,...,
3

---

Published as a conference paper at ICLR 2025
14
Agent
Agent
Q' or
Aggregation Network
B(tl-7400-1.0))
Atten(F
Agent
Self-Austion
Environment
Backbone
of
Individual Network
MA
u
Agent
of
ME
Agent
(1--7e1--1
Cof at)
Figure 3: Incorporating MA E into individual agents. Given backbone MARL algorithm, MA is integrated into the backbone individual network of each agent during decentralized execution. M serves as global information inference module by only using local observations of the current agent and inferring other agents' information.
chooses an action w' E U which consists a joint action u = {u' w²} E U", The environment follows the transition function P(s'|s, s, u) S U S + [0. 1]. All the agents share the common reward function r(s, u) x U + R. Each agent only obtains an individual partial observation of E Z with the observation function ((s, i) S N Z, and has
ieservation-action history 7' E T = (U) and an individual policy n (w'(=') TxU + [0, 1]. The objective is to maximize the expected return Es, Lioe 1m== r++ (8t, Ur] with discount factor 7 E [0. 1).
Masking: While random masking to states and actions commonly adopted in single agent RL (Liu et al. 2023a Wu et al., (023). our aim is infer other agents' information only with local observations hence we apply masking at the agent level. During centralized training, we can access all agents' trajectories r (-" ( - T(') where नi (oi, wf). As illustrated in Figure 2 we randomly select agents and mask out trajectories of these agents. Then MA?E learns to recover the entire trajectories from the masked trajectories that it outputs 7--441: where 7. represents
recovered trajectory for the agent at time step
In decentralized execution (for exploration during training or deployment after training), apart from the local observations and actions the agent (ri-7+1:t-1- of),, other information such as observations from other agents cannot be obtained. The agent only utilizes its local observations for the action selection and MA infers global information from local observations by restoring masked areas during execution, depicted in Figure
3 The comparison between agent level and random masking strategies can be found in Section 4.4 and Appendix
The detailed architecture of MA'E is described Figure MA?E follows an auto-encoder architecture (Bank et al.) 2020), with the encoder and decoder structured similarly to Transformer (Vaswani et al.) (017), using Multi-Head Attention (MHA) and feedforward networks. The decoder does not use masked MHA. Instead, a separate layer is utilized for masking before the input data is fed into the encoder and decoder. In other words, observations and actions are embedded in the embedding layer, undergo masking and positional encoding layers, then are fed into the encoder and decoder. For positional encoding, the orders of both time steps and agents are valuable information as MA²E takes histories of multiple agents. Therefore, apply positional encoding considering Agent and Time dimensions, and the related results are presented in Section 4.4 and Appendix D
Architecture:
Incorporating MA 'E into Individual Agent: MA 'E is integrated into individual agents as illustrated in Figure Following this procedure, note that MA²E 2E can be easily plugged into existing MARL algorithms. Depending on the selected MARL algorithm, each agent has a backbone network E corresponding the value function Q' policy म.. MA'E is integrated internally within each agent
4

---

fed into the aggregation network. Finally, the aggregation network produces the individual value Q' Published as a conference paper at ICLR 2025
in addition to E. Each agent only takes its own local observations to determine actions; hence, inputs for both E and MA²E are also local observations. Then MA²E reconstructs trajectories of all agents
from its local trajectories, where the output for the agent at the current time is replaced by (o). k.) since of is already observed. The recovered trajectory for the current time i, 7/ = (7)) Tr is fed into the self-attention layers. The self-attentio layers use self-attention mechanism (Vaswani iet al., 2017) to selectively focus on other agents' information for the agent's 's current situation Tt. For the agent and j. the attention weight Wa.j is calculated as:
expli, WT 7
W E+I. -- exp WT)
W3 and Wa are weight matrices such as fully connected layers for the query and and key k. respectively. Then glevance-weightee value is computed by taking the weighted sum of the attention weights and values as shown below:
++1,
Allen, (7) (T)
Σ
Wi W W
where W,, is another weight matrix. From the backbone network E,, we take the outputs before the last layer E(ri 1(72-1+1=-1-0) o,) latent variables, and both E(ri-T+1--1: of) and (7!!
or policy x' from which the agent chooses the action u'. This framework enables fully decentralized execution for each agent without access global information.
The training of MA is divided into two stages: MA pre-training and MAAE fine- tuning. During the pre-training stage, MA 'E is trained using samples collected by a random policy before the policy training begins. If we train MA from scratch and agents' policy concurrently, the generated information from MA would be a noise that interferes with the policy training. Hence, the pre-training is needed to ensure that MA can properly infer unobserved information. The objective of MA defined in Eq. (3). is to minimize the Mean Squared Error (MSE) loss between masked and true histories. Pre-traiming continues until the loss falls below specified threshold.
Training:
1 nT
L MAPE
( -7)2
After pre-training of MA the policy training begins. In this stage, fine-tuning of MA also takes place. However, the policy is independently trained according the chosen MARL method, and M is periodically updated with the loss defined in Eq. (3) after updating the policy sufficient times. Both the agents' policy and MA are trained using collected trajectories in replay buffer.
EXPERIMENT AND RESULT
Environment and Setup We evaluate M on the StarCraft Multi-agent Challenge (SMAC) (Samvelyan et al.) 2019). SMACv2 (Ellis et al. 2023). and Google Research Football (GRF) (Kurach al.] 2020) environments. SMAC is one of the most popular MARI. benchmark, which covers wide range of cooperative microcontrol scenarios. We conduct the experiments on SMAC HARD and SuperHARD scenarios. SMACv2 complements the deterministic property of SMAC by randomizing start positions and unit types, and changing the units' sight and attack ranges. In GRF, multiple agents cooperate to play a football game. All the experiments are conducted during 2 x 10 time steps for each run, and we report the average win rates with the shaded standard error from three different random seeds. More experimental details can be found in Appendix A
4.1 IMPROVING PERFORMANCE BY INCORPORATING MA 'E INTO MARL ALGORIHTM
Firstly, we investigate the effectiveness of integrating MA 'E into existing MARL methods to improve the performance. We employ fine-tuned QMIX (Hu et al. 22221). which achieves SOTA performance through parameter fine-tuning from the vanilla QMIX (Rashid et al. 2018). as backbone algorithm apply M We compare M with various MARL algorithms including VDN (Sunehag et al.
5

---

Published as a conference paper at ICLR 2025
VON
DMX
DIV-ONI CIAOMI
OPLEX
CCLAA
NARCL
6n1, VV_
corridor
MMMMZ
2.0
1.0
9.6
0.6
We
0.4
0.4
0.4
0,00
08.0
0.00
Time staps (3x6)
Time staps (1ed)
Time staps (led)
6mm
0.8
0.8
0.5
0.0
0.5
0.2
0.00
0.2
0.4
0.5
0,5
1,0
0.50
0.5
1.0 1.5
2,0
%
0.5
3.0
3.5
2.0
Figure 4: Performance comparison with baselines in SMAC scenarios. The blue line represents the model where MA?E is plugged into QMIX, while the other colored lines correspond to baselines. The proposed model performs better in both HARD and SuperHARD scenarios in SMAC.
OMIX
OW-OMIX
CW-OMIX
OPLEX
COLA
10gen terran
10gen_protose
10gen, zerg
1.0
3.0
1.0
0.8
0.6
0.8
Rate
Auweage Balte
0.6
0.5
0.6
0.4
00.0
2.0
00.0
2.0
0.
1.0
Time
Time
(tet)
Figure 5:
Performance comparison with baselines in SMACv2 scenarios. The blue line represents the model where MA is plugged into QMIX, while the other colored lines correspond to baselines.
2017 QMIX (Rashid et al., 2018 OW-QMIX (Rashid et al. 2020). CW-QMIX (Rashdd et al.
2020j, QPLEX (Wang et al. 2020), MAC2L et al. 2023), COLA (Xu et al. 2023) as baselines.
The performance comparison between the model with MA?E and the baselines is illustrated in Figure 4 and Figure 5 QMIX+MA'P is our proposed model, which is fine-tuned QMIX (Hu ct al. 2021) with the addition of MA In the HARD and SuperHARD scenarios of SMAC and in SMACv2, QMIX+MA exhibits higher sample efficiency and superior win rates compare to all baselines. In scenarios with higher difficulty levels, such corridor and 6h_vs__ _2, the performance difference between the proposed model and the baselines becomes more pronounced compared relatively easier scenarios like 2c_vs__ vs_ 64zg. Specifically, in challenging scenarios such as corridor and 6h_vs _________,____ baselines struggle to achieve victories even after two million time steps, whereas QMIX+MA attains high win rates. In addition, even when compared QMIX, which is fine-tuned version showing state-of ofte--of-the-ar art performance in MARL, the proposed method shows superior performance. Even the 1 10gen__teeeeeeeeeeeee terran scenario in SMACv2. QMIX+MA demonstrates higher performance compared to the baselines.
4.2 COMPARISON WITH THE FULL STATE AND COMMUNICATION METHODS
We evaluate whether MA can accurately infer full observations from partial observations. We compare the results of the model when using full state with the results of the model using MAZE.

---

Published as a conference paper at ICLR 2025
GMIX k.l state (61208)
OMIX GNIX+MIME (oUr5)
vs, 32
3G VS 52
2c VS. 642g
corridor
1.0
1.0
0.8
1.55
0.5
Evengee Vite Bane
Anerzgn Whh
0.6
Avengge
0.4
13
02
0.
0.
3,30
0.2
3.0
0
0.2
3.0
00.0
0.4
0.5
0.00
20
1led)
Time
Time (ted)
Figure 6: Performance comparison between using full observation and utilizing MA in different SMAC scenarios. Green line represents the case where QMIX is used with the full state, the blue line corresponds to our proposed model, and the red line represents the fine-tuned QMIX which is baseline model in the experiment. The win rates of the model using the full state and the model with M applied exhibit similar patterns.
Table I: Comparison of MA?E observation inference accuracy based on training progress
Before Pre-Training M
After Pre-Training MA-E
After Fine-Tuning MA EE
Type
True Allies
True Enemnies
Inferred
Allies
Enemies
Discrepancy
Distance (L2 norm)
0,9933
0.3989
0.3588
Health (1.1 norm)
0.6606
0.5585
0.4766
All Info (L.1 norm)
0.5661
0.3584
The fully observable model replaces the MA of individual agents which is Figure 3 with full observation and utilizes the output obtained through the self-attention layer. The results are depicted in Figure 6 The comparison across four SMAC scenarios shows a similar pattern between using full observation and using MAÉE. Moreover, it evident that the performance is better when using full observation using MA compared to using only partial observations. Especially in scenarios like 3s_vs_52, where inter-agent information sharing is crucial and the difficulty is higher compared to other scenarios, the differences become more pronounced.
Table D compares the inference accuracy according to the training progress of MA It shows the results of agent 1 inferring the observations of agent 2 in the 3s_vs_55 5z scenario in SMAC. From left to right in the table, the results are from models at progressively advanced stages of training, and the figure displays the inferred relative positions of agents from the observation inference results of other agents, plotted on a two-dimensional plane. In the figure, red and blue icons represent true values and the inferred values, respectively, indicating that learning progresses, the inferred values become closer to the actual values. In addition, the values in the table mean the differences between the actual values and the MA?E's deduced values for position, health, and the entire observation values of other agents. The discrepancies decrease as MA?E learns, demonstrating that MA can infer values closer to the real values through training. Based on the results, we can conclude that MA?E can successfully infer information similar to full states.
Another way to utilize information over partial observations is information exchange among agents through communication. We compare MA?E with communication methods in MARL domain including MAIC (Yuan et al. 2022), QMIX att (Hu et al., (021)) and CommFormer (Hu et al. 2024). The learning curves can be found in Appendix E As shown in Table 2 QMIX+MA' E demonstrates performance that is comparable to or better than communication methods. Since MA'E does not rely on direct information exchange, its performance highlights the strengths of our method.
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of MA²E with communication-bnn based methods
Scenario
Steps
QMIX+MA (ours)
MAIC
QMIX-at1
CommFormer
3s_vs_$a
2M
0.99 + 0.01
0.86 + 0.05
0.41 + 0.37
0.06 + 0.04
2c_vs_642g
IM
0.97 + 0.03
0.76
0.34
0.59 + 0.06
Sm_vs_ 6m
2M
0.48 0.08
0.69
0.81 - 0.04
0.61 + 0.05
corridor
2M
0.91 + 0.10
0.0
0.30
0.0 - 0.0
6h_vs_88
2M
0.53 + 0.08
0.0
0.04
0.0
MMM2
2M
0.79 0.08
0.97 0.07
0.20 0.08
0.0
swP.rtPPronn VOM
SRLE-OREES IIPLEXX
MA Teans
WIT
2c_v5_60g
2c_vs_64rg
2c_vs v5_64g
2.0
1.0
agaage Wi Pott:
2-6
0.8
62
00.00
0.2
2.4
3.6
0.8
3.0
08
0.2
2-4
0.6
0.6
10
0.30
0.2
04
08
3.
creps
10gen_terne _teman
Leeran
10gen_terre
0.8
0.8
0.4
0.8
Beoaage We
0.
0.4
04
0.2
0.2
09.0
1.0
2.0
00.0
0.5
3,5
04.
855
cteps (1w6)
CEPDC (1w6)
Tiue deppe (1e6)
Figure 7: Performance comparison between the model incorporating MA in QPLEX and VDN and baselines in SMAC and SMACv2 scenarios. (Left): Comparing the performance of the model with MA applied to VDN and the performance of VDN. (Middle): Comparing the performance of the model with M applied to QPLEX and the performance of QPLEX. (Right): Comparing the performance of the model with MA applied RIIT and the performance of RIT to assess whether M leads to performance improvement when applied to policy-based method.
4.3 EXTENSIBILITY AND TRANSFERABILITY OF MA
To explore the applicability of MA to various MARL algorithms in general context, conduct experiments by applying MA to various value- based models such as VDN and QPLEX. The exper- imental results are as shown in the Figure 7 The re- sults across scenarios both SMAC and SMACv2 indicate that the addition of MA 'E improves perfor- mance compared the baselines. MA not only enhances sample efficiency but also converges to higher win rates in fewer time steps.
MATEC VOhiout
Transfered (DNjoo)) VDM
5m, Us.
HHH2
1.0
0.6
0.8
$.50
0%
steas 21050
lane s1000 Ctet)
Furthermore, to confirm the applicability of MA?E to policy-based algorithms, we conduct experi- ments by incorporating MA?E into the policy- based algorithm RIIT (Hu et al., 2022I. The right figures in Figure 7 illlsttrate that RITT with M
Figure 8: Performance comparison between transferred MA and default MA in differ- ent SMAC scenarios.
exhibits better performance compared to the baseline models. The experimental results demonstrate that MA can be seamlessly integrated into both basue and policy-based algorithms.
In order to confirm whether MA can be transferred to other backbone networks, MA is first trained with QMIX. Then the trained MA?E is transferred to the VDN network and only MA²E
8

---

Published as a conference paper at ICLR 2025
VDN+MW-E(UurR)
VON
Acodbmy. 2_ (_wth_ keeper
scademyyrounterttted www.terataccc.ccoooo eosy
Accodmyy_couutsrrttaa wwt._ooouoneeeec..cc nard
ving
Awerage Vin rolo
R00
Aedrago
-4
Speregge
4
004
0.
0.0
20
4.0
5.0
8.0
00
20
4.0
6.0
8.0
0.0
20
4.0
6.0
5.0
10.0
Training steps
Training steps
Traniig steps
Figure 9: Performance comparison between the model incorporating MA in VDN and baselines in Google Research Football scenarios
finetuning is performed as training progresses, without pre-training. Figure 8 compare the trans- ferred MA E with the M that goes through the pre-training process. The model equipped with transferred MA " performs better in the HARD scenario and shows comparable performance in the SuрerHARD scenario, which demonstrates the transferability of MA?E.
To further verify the effectiveness of MA?E across diverse environments, additional experiments are conducted in GRF We apply M to VDN. As in Figure 9 integrating MA?E significantly improves performance in three different scenarios. Consequently, the above results demonstrate the extensibility of MA?E for various MARL algorithms and domains,
We evaluated the performance of VDN, QPLEX, RIIT, and another policy-based algorithm, IPPO (de Witt et al. 2020) with MA
in diverse scenarios and the results are illustrated in Appendix performance across different algorithms and tasks.
C
MA'E consistently improves the
4.4 ABLATION STUDIES
To identify suitable structure and byyer-parameete of MA?E, we conduct ablation studies across var- ious settings. The comparison conducted the vs 5z vs__v scenario SMAC. The experimental results for comparison are illustrated in Figure TO and in Appendix D
Masking Strategy: We compare two masking strategies: agent- hased and random masking. Random masking removes random portion of the input data, irrespective of the agents. The masking ratio is randomly chosen from 0.15, 0.35, 0.5, 0.75, and 0.95, with the masking point also being randomly determined. This ratio follows the suggestion from Liu al. (2023a). Figure T0 (a) compares win rates based on different masking strategies and agent- masking is superior because it can effectively capture correlations between agents.
The Number of Trajectories: To test the appropriate trajectory length, we conduct a comparison of the number of trajectories in five different settings: 3, 5, 7, and 9. Figure TO (b) compares perfor- mance based on the number of trajectories. The performance does not show a consistent difference among the different number of trajectories. However, can be observed that the performance best when the number of trajectories is 5, and it deteriorates as the number becomes smaller or larger than 5, Therefore, to achieve good performance with MA E, is necessary set appropriate number of trajectories depending on the scenario or environment.
its
To test the effect of the positional encoding design, we compare positional en- coding using three different encoding methods Agent, Time, and Both. Agent encoding distinguishes encoding based on agents, while Time encoding sets different values based on the trajectories. In the Both setting, encoding takes both Agent and Time into account. For example, half of the embedding space is dedicated to agent-based encoding, and the remaining half is utilized for time-based encoding. Figure [0 (c) compares performance based on the positional encoding design. As we can see, the performance is significantly better when considering both time and agent, as opposed to either of them individually. When considering only one of either time agent, there is not significant difference in performance compared to the case where MA E is not used. In particular, when considering only time, there are intervals where the performance is even worse than without MA?E
Positional Encoding:

---

Published as a conference paper at ICLR 2025
Randors
DNO
QMX
Agent Tart Thow CMI
MAREHOVIX
1.0
3t ve_ So
10
3v vs
1.0
36 vs. S0
3a. vS. B2
0.0
00
0.3
TTanniggkkp
1.0
15
20
00
05- Training
10
15
00
0.3 Traring skia
10
1eb
Inanrgatape
(a)
(b)
(c)
(d)
Figure 10: Performance comparison based on hyper parameter settings or strategies. (a): Performance comparison based masking strategies. The agent-based masking strategy applied model outper- forms both the random masking strategy applied model and the default model. (b): Performance comparison of MA?E based on the number of trajectories. (c): Performance comparison of MA E based on the positional encoding strategies. When considering both agent and time, it shows the best performance. (d): Performance comparison of MA with or without pre-training and fine-tuning.
MA`E Training: As mentioned in Section 3 MA'E training is divided into two stages. To verify the importance of both stages, we report the performance of MA E without pre-training (w/o PT) and without fine-tuning (w/o o FT). As shown in Figure I0] (d), employing both pre-training and fine-tuning achieves the best performance. Moreover, there is tendency for the performance of the case without pre-training (w/o PT) to be better than the case with pre-training only (w/o FT), but the variance of w PT is very high. This indicates that when MA²E is underfitted and its outputs used in decision making are likely to act as noise. Therefore, pre-training is necessary to reliably improve the performance and fine-tuning is required to align MA 'E with the policy.
5 CONCLUSION AND LIMITATIONS
In this paper, we propose novel method MA²E, which utilizes the masked auto-encoder to address partial observability in the context of multi-agent reinforcement learning. MA?E empowers agents to derive global insights solely from local information, thereby enhancing their collaborative decision-making capabilities. The proposed method seamlessly integrates with both aalue-based and policy-based MARL algorithms. Through extensive experimentation, we have substantiated its effectiveness in enhancing sample efficiency and elevating task-solving proficiency across a diverse set of scenarios within the SMAC, SMACv2, and GRF environments. We posit that the incorporation of MA²E into MARL, which extends the observational horizon of agents, stands as a pivotal advancement MARL research. By introducing a methodology that addresses the challenge of partial observability in MARL, MA has emerged as a cornerstone in the field.
Nonetheless, our proposed model exhibits limitations. When agents are positioned far apart such that there is overlap in their observations, the capacity to infer information about other agents is severely restricted, potentially resulting in the inference outcomes of MA being not useful. A more comprehensive delineation of these limitations can be found in Appendix H Additionally, the structural characteristics of MA make it relatively difficult to scale to a varying number of agents. In future work, we plan to investigate approaches that enable scalability, such team composition.
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea governmen((MSIT) (No. RS-2024-00457882, AI Research Hub Project), by Institute for Information & communications Technology Planning & (valuationnIITP) grant funded by the Korea (MVIT (RS-2019-I119075, Artificial Intelligence Graduate School Program(KAIST)) by Institute of Information & communications Technology Planning & Evaluation (IITP) under Open RAN Education and Training Program NUTP-2025-RS-2044 (RS-2024-00429088 grant funded by the Korea gOVT,, and conducted by
10

---

Published as a conference paper at ICLR 2025
Center for Applied Research in Artificial Intelligence (CARAD) grant funded by DAPA and ADD (UD230017TD).
REPRODUCIBILITY STATEMENT
For the details of environments and hyperparameters, please refer Section and Appendix A To run our method, please download the supplementary material and follow the instructions in README files. We employed pymarl2 (Hu al. 202T or the official codes from the authors for baselines.
REFERENCES
Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. CoRR, abs/2003.05991, 2020. URL 05991
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers.
arXiv preprint arXiv:2106.000 08254, 2021.
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: review of challenges and applications. Applied Sciences, 11(11), 2021. ISSN 2076-3417. doi: 10.3390/app 1 14948. URL https
/wwww. cow/ wtt.c/mmc222// 2076- 3417/11/11/4948
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels, In International conference on machine learning, pp. 1691- 1703. PMLR, 2020.
K Clark. Electra: Pre-traing tre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:201 2020.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 1010.88000 2018.
Wei Du and Shifei Ding. A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artificial Intelligence Review, 54(5):3215-3238, 2021.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Ma- hajan, Jakob Nicolaus Foerster, and Shimon Whiteson. SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning. In T'hitt--eventth Conference Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https http://wwwwnne//ooooe----------
Jakob Foerster, Ioannis Alexandros Assacl, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey, Artificial Intelligence Review, 55(2):895 943, 2022.
Cong Guan, Feng Chen, Lei Yuan, Chenghe Wang, Hao Yin, Zongzhang Zhang, and Yang Yu. Efficient multi-agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 35:1020-1033, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.

---

Published as a conference paper at ICLR 2025
Jian Hu, Haibin Wu, Seth Austin Harding, Siyang Jiang, and Shih-wei Liao. RITT: rethinking the importance of implementation tricks in multi multi-agent reinforcement learning. CoRR, abs/2102.03479, 2021. http:tts::
org/ abs / 221/22122222 03479
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. arXiv preprint wwXiv:2405.085500 2024.
Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marein Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4501 4510, 2020.
Dapeng Li, Zhiwei Xu, Bin Zhang, and Guoliang Fan. From explicit communication to tacit cooperation: A novel paradigm for cooperative marl. arXiv preprint arXiv:2304.1443 2023.
Fangchen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. Advances in neural information processing systems, 2023a.
Jie Liu, Yinmin Zhang, Chuming Li, Chao Yang. Yaodong Yang, Yu Liu, and Wanli Ouyang. Masked pretraining for multi-agent decision making, 2023b.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arxiv: 2019.
Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 721 1211-72 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatiee-compettvve environments. Advances in neural information processing systems, 30, 2017.
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges. solutions, and applications. IEEE Transactions on Cybernetics, 5019):3826-3839 2020. doi: 11109/CCYB.0200.2772..
Zepeng Ning and Lihua Xie. A survey on multi-agent reinforcement learning and its ap- plication. Journal of Automation and Intelligence, 3(2):73-91, 2024. ISSN 2949-8554. doi: Beiiene//rricee//1///99955200000
URL http::://wwwwws.cceeeeeeee..ce.e www.eceeeeeeec............. com/
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. CoRR, Abs/1803.11485, 2018. URL http://arxiv.org/
1803.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation. CoRR, abs/2006. 10800, 2020. URL https: //arxiv
org/abs/ 2006. 10800
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. Rudner, Chia-Man Hung, Philiph H. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.
Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. Advances in neural information processing systems, 32, 2019.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pp. 1332-1344. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He, and Xiangyang Ji. Self-organized group for cooperative multi- reinforcement learning. Advances Neural Information Processing Systems, 35:5711-5723 2022.
Haolin Song, Mingxiao Feng, Wengang Zhou, and Houqiang Li. Ma2cl: Masked attentive contrastive learning for multi-agent reinforcement learning, arXiv preprini arXiv:2300 Siv::2306002006, 2023.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation.
Advances in neural information processing systems, 29, 2016.
Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg. Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Valoe-decompositton networks for cooperative multi-agent learning, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: duplex dueling multi-agent q-learning. CoRR, abs/2008.01062, 2020. URL. fttpps://axxiv.or//abba. 0008.01060
Philipp Wu, Arjun Majumdar, Kevin Stone, Yixin Lin, Igor Mordatch, Pieter Abbeel, and Aravind Rajeswaran. Masked trajectory models for prediction, representation, and control. arXiv preprint arXiv:2305.02966. 2023.
Zhiwei Xu, Bin Zhang, Dapeng Li, Zeren Zhang, Guangchong Zhou, Hao Chen, and Guoliang Fan. Consensus learning for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11726-11734, 2023.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:2461 24624, 2022.
Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang, Multi-agent incentive communication via decentralized teammate modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9466 9474, 2022.
Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv preprint arX15:2203.08975, 2022.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
In this section, we introduce the environments used in the experiments, the baseline algorithms, as well as the hyperparameters and computational resources. Experiments carried out NVIDA A6000 and GTX3090 GPUs and AMD EPYC 7313 CPU.
A.1 ENVIRONMENTS
We conduct experiments in the following environments:
StarCraft Multi-Agent Challenge (SMAC) (Samvelyan al. 2019) from https://
github. com/ oxwhirl/smac which is licensed under MIT liceese.
SMACv2 Ellis et al. 2023) from
com http///whhh.... smacv2 which
is licensed under MIT license.
Google Research Football (GRF) Kurach al. 2020) from httmm//tst/ggthub...... com/
g wweesearch/oootoeee
which is licensed under Apache License 2.0.
All algorithms are implemented based on the open-source framework pymarl2 (Hu et al. 2021) from
https://githut http://hwwa..bm/./.i///////aaaaa which is augmented version of pymarl from https: wwwp.//www//////m/occaoa Both are licensed under Apache License 2.0.
A.1.1 SMAC
The StarCraft Multi-Agent Challenge (SMAC) is one the benchmarks widely utilized in research to evaluate MARL algorithms. Units from the strategy video game StarCraft II engage in confrontations with each other in diver scenarios. The objective is for multiple agents to collaborate in defeating the enemies. There multiple scenarios, each categorized into difficulty levels such EASY, HARD, and SuperHARD. We primarily conduct experiments in HARD, and SuperHARD scenarios. Table provides a detailed description of the scenarios used in experiments.
Scenario
Difficulty
Ally Units
Enemy Units
Type
2s_vs_1sc
EASY
Stalkers
Spine Crawler
micro-trick: alternating fire
3s_vs_32
EASY
3 Stalkers
3 Zealots
micro-trick: kiting
3s $s_vs_5z
HARD
3 Stalkers
5 7ealots
micro-trick: kiting
2o_vs_642gg
HARD
Colossi
64 Zerglings
micro-trick: positioning
1 Medivac
Medivac
MMM
HARD
2 Marauders
2 Marauders
heterogeneous & symmetric
7 Marines
Marines
corridor
Super HARD
6 Zealots
24 Zerglings
micro-trick: wall off
6h_vs_8z
Super HARD
6 Hydras
8 Zealots
micro-trick: focus fire
Medivac
Medivac
MMM2
Super HARD
2 Marauders
3 Marauders
heterogeneous & asymmetric
Marines
Marines
Overload
10 jo_2r_vs_42
Roaches
communication
2 Roaches
Table 3: A detailed description of the SMAC scenario used in the experiment
A.1.2 SMACV2
SMACv2 is proposed address the shortcomings of SMAC, particularly in terms of its lack of stochasticity and partial observable characteristies (Ellis et al. 2023). Therefore, SMACv2 differs from SMAC in three main aspects.
14

---

Published as a conference paper at ICLR 2025
First, the unit composition is randomly determined. In SMAC, the generated units fixed, whereas in SMACv2, different types of units are randomly generated based on probabilities. The second difference lies in the observation probability of agents. In SMAC, when agent observes enemy, other agents can also observe the same enemy simultaneously. In contrast, in SMACv2, if one agent observes enemy first, other agents within their observation range may identify the same enemy, even if it is present. The last distinction involves adding randomness to the location where units spawned. The location where units are spawned is determined by one of two types: one is surround and the other is reflect. surround entails the creation of units in a formation where allied units surround enemy units, while reflect involves units being spawned in facing and confronting manner.
(a) surround
(b) reflect
Figure 11: Two different types of start positions in SMACv2
We conduct experiments in SMACv2 under conditions that encompass all three aforementioned aspects, and additionally, we fixed the last condition to investigate performance differences in MA based on the spawning locations. To elaborate further, when comparing the results between the surround and reflect, we assumed that surround where agent observations already sufficiently overlap, would render global information less significant. agents are spawned at appropriate distance, allowing for the assumption that through the inference of global information, they could acquire useful information about agents at long range.
the other hand, in the case of reflect,
Figure I provides examples of the reflect and surround positions. The light gray circles represent the observation range of allied agents. In the case of surround, most areas overlap. In contrast, reflect shows relatively smaller overlap compared to surround. Therefore, we believe that MA 2 can better infer global information the reflect position than in the surround position.
A.1.3 GOOGLE RESEARCH FOOTBALL (GRF)
Google Research Football (GRF) is one of the widely used benchmark in multi-agent reinforcement learning research. Multiple agents cooperate to play a football game GRF and GRF offers various scenarios. We experiment with our model in three different scenarios of GRF.
academy 3_vs_____________________ wwth keeper: The objective is for three allied agents to score goals in soccer half-court field against an opposing goalkeeper.
academy aeaadmmy_cuunterunne counterattack_eee easy: The scenario consists of four allied agents, one opposing agent, and one goalkeeper. The objective is for the allied agents execute counter-attack.
caaileemy_ccuuttrattakk hard: The scenario consists of four allied agents, two opposing agents, and one goalkeeper. The objective is for the allied agents to execute counter-attack.
A.2 HYPERPARAMETERS
The hyperparameters for the baselines in experiments listed in Table The value-based algorithms include VDN, QMIX. QPLEX, ow-QMIX, ew-QM QMIX, while the policy-buce based algorithms include IPPO, RIIT. Moreover, the hyperparameters for MA 2E are listed in Table
15

---

Published as a conference paper at ICLR 2025
Table 4: The hyperparameter settings for the bascline algorithms
Algorithms
Value-based
Policy-based
Optimizer
Adam
Adam
Batch Size
128
64, 32
TD(A)
0.6
Learning Rates
0.001
0.0005, 0.001
Replay Buffer Size
1000
64, 128
Anneal Steps
100000
100000
Gamma
0.99
Table 5: The hyperparameter settings for MA B
Hyperparameters
Value
Batch size
32
Input embedding
The number of heads
4
The number encoder layer
The number of decoder layer
2
Steps for fine tuning Pretraining threshold
500
0.015
B PSEUDOCODE
Algorithm Model with Multi-Agent Masked Auto-Encoder (MA?E) Applied
1: Initialized value networks Qo and Qo or policy network np
2: Initialized MA parameters 0 ma?e
3: Prepare replay buffer D
4: repeat
5: Run episodes through random policy and store trajectories in Buffer D
6: Update MA parameters Omate using samples in D I MARE of Σn E (
7: until When training has been done for specific number of steps or when the loss is lower than the threshold
8: Reset replay buffer
9: repeat
10:
for each episode do
11:
Get initial state
12:
while episode is not terminated do
Sample actions a, from Q with e greedy or policy me Execute actions and observe reward
14:
Store transition (St, O,, rt) in buffer D
16:
end while
17:
Update value networks Qo and Qu or policy network me Update MAdE parameters 0 Omaee
18:
19:
end for
20:
until reaching maximum total environment steps
16

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RESULTS OF MA?E WITH VDN, QPLEX, RITT AND IPPO ACROSS DIFFERENT ENVIRONMENT
We provide additional experimental results from SMAC, SMAC2, and GRE, All results were tested with random seeds.
Pritass S
W
WATERORLERIUUS) CPLEX
MWP (WFEHilT
MA90
15000
MATE-TTO (aun)
PTO
Figure 12: Performance comparison in SMAC and SMACv2 scenarios
D ADDITIONAL ABLATION STUDY
We provide additional ablation study in cc_vs_6420 similar Figure TO We can draw a similar conclusion that: using agent-level masking, an appropriate number of trajectories, positional encoding considering both agent and time, and both pretraining and fine-tuning yield better results.
Agent
Ze_ 642g
642g
2C_v5.642g
2c_v5_60 6426
3.0
Arxmge Mr Eo0
0.6
0.6
0.0
0.6
0.4
0.4
0.4
06
0.2
0.4
3.5
0.0
3,0
0/
0.2
2/4
0.6
0.0
3.3
0%.0
0.2
0.4
0.8
08
1.0
0/
0.2
84
0.6
0.6
3.0
stops
Figure 13: Additional ablation study in 2c_ _c_vs_642g scenario. Along with Figure a similar conclusion.
10
draw
17

---

Published as a conference paper at ICLR 2025
E LEARNING CURVE OF TABLE 2
We provide learning curves of each algorithm in Table 2 outperforms the baselines in most scenarios.
As shown in Figure 14 QMIX+MA
QMIX-att
MAIC
CommFormer
0.8
0.8
0.8
Actroge Wh 8322:
Aecroge Wh Rats
0.66
0.4
0.4
0.2
0.2
0.00
1.5
0..0
0.5
0.0
0.30
0.5
2.0
1.5
2.0
Time steps (3ed)
Time staps (3e6)
Time steps (3ed)
corridor
6h_ 1r__v__0r Oz
MMM2
0.0
0.0
0.0
Acorgge We
0.4
0.4
0.4
0.00
0.5
3.0
1.5
2.0
0.00
0.5
3.0
1.5
2.0
09.0
0.5
1.0
3.5
2.0
stops (305)
stops (306)
steps (1e6)
Figure 14: Learning curves of each algorithm in Table
F COMPARISON OF TRAINING COST AND MODEL SIZE WITH AND WITHOUT USING MA`E
3s vs _s_z
MAZE+QMIX QMIX
Difference
Time for pretraining Time to convergence Steps convergence Steps per second
0.77h
+0.77h
17.05h (faster)
18.22h
-1.17h (93%)
2.03M (faster)
3.52M
-1,49M (57%)
33.07
53.66
-20.59
The number of parameters
0.43M
0.12M
0.31M
Table 6: Comparison of training and execution time and the number of parameters for each method in the SMAC 3s $s_v__________________________ _s___2____ ___ scenario. The values in parenthesis represent the relative ratio between MA2E+QMIX and QMIX.
2c_vs. ___.v_66460 64zg
MAZE+QMIX QMIX
Difference
Time for pretraining Time to convergence Steps to convergence
0.84h
+0.84h
7.97h (faster)
8.76h
-0.79h (90%)
0.639M (faster)
1.021M
-0.382M (63%)
Steps per second
22
32
-10
The number of parameters
0.98M
0.65M
0.33M
Table 7: Comparison of training and execution time and the number of parameters for each method in the SMAC 2c_vs_64 .64zg scenario. The values in parenthesis represent the relative ratio between MA2E+QMIX and QMIX.
18

---

Published as a conference paper at ICLR 2025
Surround
reflect
Averoge Wi 130
0.8
0.6
0.6
0.0
0.
00
Training
1.0
1.5
0.0
Training slaps
1.0
1e6
Figure 13: Performance comparison according to the different starting position in SMACv2
G ABLATION STUDIES ON OBSERVATION OVERLAP
SMACv2 offers different starting positions based on the configurations, with two main strategies for positioning: surround. where the units are placed position surrounded by the enemy, and reflect, where they are positioned head-on in a confrontation with the enemy. The extent of overlap in agent observations varies depending on each position. (For more details, please refer Appendix A.1.2 We utilize these two positions to compare the performance of MA 'E based on the extent of observation overlap. In both cases, the performance better than the baseline, but it particularly evident from Figure 15 that the performance is significantly better in the reflect compared to the baseline. Therefore, can be concluded that when the observation range overlaps at a reasonable proportion, MA E can better infer global information, opposed to cases where the overlap is too extensive.
H LIMITATIONS
MATE BMll-AUleiiinnonmmeeeeiennn
QMIX
1o 2r vs_ 4r
0.8
Average sin
0.00
0.25
0.30
0.75
1.00
1.25
1.50
3.75
2.00
Training steps
(a) A Screenshot of 10. 2r_ 4r scenario in SMAC
(b) Win rate comparison between using communica- tion and using MA-E,
Figure 16: A scenario where MA fails to infer global information
agent must use observations clues to infer the actions or movements other agents. Therefore, if there is no information about other agents in the observation of an agent, MA?E cannot effectively capture the information.
For example, in environments where agents are spread far apart due to large map size, and their observations do not overlap, MA?E may struggle to infer the information. Figure 16a shows a screenshot of the 10_22_2222 2r vs 4r scenario, and Figure [6b] shows comparing the win rates when the model uses communication and when the model uses MA?E. In this map, enemies and allies spawn random locations among the four corners (top-left, bottom-left, top-right, bottom-right) of
19

---

Published as a conference paper at ICLR 2025
the map. Allied agents consist of two attacking units and one observation unit, with the observation unit spawning at the same location as the enemies.
If communication is possible, the observation unit can inform the attacking units of the enemy's location. On the other hand, since the observation unit is outside the observation range of the attacking units, they cannot infer information about the observation unit. We compare our proposed model with the communication method presented by Hu et al. (2021), called QMIX-Attention, to investigate this situation. As shown in the Figure 16b the performance of our proposed model is relatively lower compared to the case where communication is used. Nevertheless, despite that, it can be confirmed that using MA E does not make lower performance compared the default model.
(MIrs
Commnet
G2ANet
MAIC
QMIX-att
QMIX
38
10. 2r vs 4r
1.0
1.0
Average win rate
Average win rate
0.6
0.4
0.4
0.2
0.0
2.0
0.0
1.0 Training steps
2.0
Training steps
166
1e6
Figure 17: Performance comparison with communication based MARL algorithms in two SMAC scenarios
We also compare with communication methods Commnet (Sukhbaatar et al. 2016). G2ANet Liu
al. 2020), MAIC Yuaan et al.) (022),, and QMIX-att (Hu et al.J 2021). The results are shown in Figure 17 In 3s vs 5z, appropriately overlapping observations allow MA to effectively infer information, and the use of attention layers reduces the space size. Conversely, communication based methods increase the space size and require communication overhead, leading to slower convergence and sample inefficiency. However, 1o__r____ 2r__v_______ s__s__4x MA struggles to perform effectively due to very little observation overlap among agents, while communication methods (QMIX-ati and Commnet) can still fully share the information hence achieves better performance. Thus, we note that the need for communication vary depending the environment.
I Is MA E PERFORMING WELL BECAUSE IT IS OVERFITTED TO THE ENVIRONMENT?
M learns the dynamics of the environment during the training process and captures global information through its ability to infer masked areas. Therefore, from one perspective, it can be considered that M performs well due overfitting to the environment. Specifically, in the SMAC environment, there is a significant performance difference between algorithms applying MA ZE and baseline, but in the SMACv2 environment, this difference is not as pronounced. This suggests that MA may be overfitting to the environment because SMAC is more deterministic compared to SMACv2, making easier for the model to overfit.
However, the issue of not performing well in SMACv2 is common difficulty not only for MA?E but also for existing MARL algorithms. In reflecting the stochastic characteristic of SMACv2, the random properties
generate scenarios that unwinnable from the start turn each episode into a completely different task, requiring a multi-task approach. Therefore, it is difficult to argue that M is overfitting just because it does not perform well in SMACv2.
Nonetheless, to prove that MA E is not overfitting to the environment, we created a non-deterministic scenario by adding random characteristics to the 3s vs _z scenario of SMAC and used it for
20

---

Published as a conference paper at ICLR 2025
(ours)
1.0
0.8
Average win rate
0.6
0.4
0.0
Training steps
(a) The spawn locations of enemy and ally agents in the modified _______22 scenario.
(b) Win rate comparison between using MA 'E and without MA 'E case
Figure 18: The experimental results in the environment with added stochastic characteristics
experimentation. In the original 3s vs 5z scenario, the positions where enemy and ally agents spawn are fixed. In contrast, in our modified scenario, the enemy and ally agents spawn at random locations within the blue area shown the Figure 18a Thus, stochastic property is added where the spawn positions of the enemy and ally agents change each episode.
The experimental results shown the Figure T8b
As result, it be that the model using MA significantly outperforms the model without it. Therefore, it can be concluded that MAÉE does not overfit to the environment but appropriately infers global information according the situation.
J BROADER IMPACT
Our study introduces Multi-Agent Masked Auto-Encoder (MA²E) to enhance decision-making in multi-agent systems. By addressing partial observability through masking perspective, our approach has broad applications in real-world scenarios such as military operations, autonomous driving, traffic systems, and robotics, promising improved decision decision-making makiing across diverse multi-agent environments. While explicit communication can selectively send the messages, MA is trained with full information. Hence the proposed method may cause privacy or security issues in some real world applications, which would be mitigated by anonymizing data.
21


Reviews:
[{"review_id": "HHdawkqW1S", "reviewer": "Reviewer_dk6X", "summary": "Despite the advancements achieved with CTDE, partial observability remains a barrier to effective cooperation among agents. This paper's approach is to enable each agent to infer global information solely from its local observations, allowing it to gain a comprehensive view without relying on communication. To achieve this, the authors introduce the Multi-Agent Masked Auto-Encoder, which leverages the masked auto-encoder architecture to infer the information of other agents from partial observations. By using masking to learn how to reconstruct global information, the method functions as an inference module for individual agents within the CTDE framework. The method can be integrated into existing multi-agent reinforcement learning (MARL) algorithms and has been experimentally shown to enhance performance across a variety of environments.", "questions": "- CDS [1], a well-known MARL paper, has reported a better performance (than MA$^2$E), using the same number of training time steps, in super-hard corridor, MMM2 and 6h vs 8z tasks, in which this paper reports a \"more pronounced improvement\" over baselines. Therefore, it would be beneficial if the authors included in the comparison a few exploration-based state-of-the-art methods (such as CDS) which have shown remarkable performance in SMAC tasks.\n- In execution time, for agent $i$, the paper says that MA$^2$E takes only the local information of agent $i$. However, in Fig. 2, it is illustrated that the model should take as input information of all agents. So, in execution time, does the model simply masks the positions (trajectories) of the other agents?\n- Considering the strong performance of the proposed method, as compared with the communication-based methods, have the authors compared the method with the communication-based baselines in Traffic Junction benchmark (which provides very challenging tasks aiming to test the effectiveness of agent communication)?\n\n[1] Li, Chenghao, et al. \"Celebrating diversity in shared multi-agent reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 3991-4002.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an important problem in cooperative MARL, that is dealing with partial observability without communication.\n- The paper proposes an interesting, novel method which achieves very good performance in challenging SMAC and SMACv2 tasks, outperforming well-known MARL baselines, such as QMIX, QPLEX, using a relatively small number of training timesteps.\n- The proposed method outperforms methods based on similar approaches, such as MAC2L [1].\n- Interestingly, the proposed method can achieve comparable performance, or even outperform state-of-the-art communication-based methods.\n\n[1] Song, Haolin, et al. \"MA2CL: masked attentive contrastive learning for multi-agent reinforcement learning.\" Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.", "weaknesses": "- The framework requires a pre-training stage, which can be impractical for certain scenarios. How much pre-training is needed? How extra wall-clock time is needed for the pre-training?\n- The paper conducts a limited number of experiments in GRF (that is, only for a comparison with VDN). \n- The presentation in the incorporation of the method into the MARL backbone algorithm can be improved.\n\nThe authors have addressed my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vAEj3fJjmK", "reviewer": "Reviewer_GwqF", "summary": "This paper introduces the Multi-Agent Masked Auto-Encoder (MA$^2$E), which leverages the masked auto-encoder architecture to infer information about other agents from partial observations. Specifically, MA$^2$E masks out the trajectories of other agents at random, enabling the agent to non-centrally reconstruct information about other agents based solely on its own local observations. This approach enhances sample efficiency and final performance during training in Dec-POMDP settings. The authors conducted extensive experiments in the SMAC, SMACv2, and GRF environments, demonstrating the outstanding performance of MA$^2$E.", "questions": "1. Why did you choose to use trajectories from multiple time steps as input for the masked auto-encoder instead of the hidden states outputted by the RNN in the agent network?\n\n2. SMAC has many representative scenarios, such as 8m_vs_9m, 10m_vs_11m, 27m_vs_30m, and 3s5z_vs_3s6z. Why were these scenarios not included or showcased in the experimental section?\n\n3. Table 2 only presents the final convergence results of communication-based methods. Could you also provide their learning curves?\n\n4. Similarly, regarding SMACv2, the paper only displays results for 10gen_terran (Terran_5_vs_5) and Protoss_5_vs_5. Can you include additional results from other scenarios?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide the necessary code and hyperparameter details. A preliminary review did not reveal any issues. Sharing the original code significantly enhances the credibility of the experimental results presented in the paper.\n\n2. The motivation behind the paper is clearly articulated: the goal is to non-centrally reconstruct information about other agents using only the local observations of the agent itself, thereby improving sample efficiency in training. Although the method is straightforward, the experimental results reveal significant performance differences between MA$^2$E and other algorithms, particularly in super hard scenarios.\n\n3. The selection of baselines in the experimental section is representative, incorporating many classic and relevant algorithms. Additionally, the experimental environments are diverse, including SMAC, SMACv2, and GRF.\n\n4. The ablation studies are comprehensive, covering nearly all relevant variations. Furthermore, the visualization of observation inference accuracy is effectively executed.", "weaknesses": "1. There are several typos throughout the paper. For instance, a sentence is repeated in line 255, and there is an unnumbered figure in line 907.\n\n2. MA$^2$E employs a two-stage rather than an end-to-end training approach, which contrasts with many existing works. While the authors demonstrate through ablation experiments that end-to-end training could lead to decreased performance, the pre-training aspect of the masked auto-encoder should still be factored into the algorithm's sample efficiency.\n\n3. Based on the experiments, MA$^2$E still shows a significant correlation with observation overlap, indicating that communication-based methods cannot yet be fully replaced by MA$^2$E.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pu41kQ2P9r", "reviewer": "Reviewer_bnEg", "summary": "This paper presents Multi-Agent Masked Auto-Encoder (MA2E), which utilizes the masked auto-encoder architecture to infer the global state from partial observations for MARL. MA2E is trained by masking the agent-level information to recover the entire trajectories. Experiments on SMAC, SMACv2, and Google Research Football environments show that MA2E achieves superior performance when compared with representative MARL methods.", "questions": "1.\tWhy the results of COLA and MA2CL are missing in 10gen_terran and 10gen_protoss in Figure 4?\n2.\tCould the authors also compare with MaskMA? Could MA2E be extended into MAPPO and MADDPG?\n3.\tHow does the sight range of each agent affect the performance of MA2E?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe motivation for using MAE to infer the global state from agent observations is straightforward and well-motivated.\n2.\tThe performance of the proposed MA2E is superior across different environments.\n3.\tThe authors compare MA2E with other works using masked modeling such as MA2CL.", "weaknesses": "1.\tAlthough it is claimed that MA2E can be easily integrated into existing MARL algorithms. Only QMIX-style algorithms are tested.\n2.\tMA2E is difficult to scale to a varying number of agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ode7rDDYtw", "reviewer": "Reviewer_2kwB", "summary": "This paper focuses on multi-agent reinforcement learning under partial observability and proposes to reconstruct the agents' trajectory information through a Multi-Agent Masked AutoEncoder (MA$^2$E). During execution, the agent inputs its trajectory into MA$^2$E to infer other agents' trajectories, which is then processed by a self-attention layer. Finally, the agent aggregates the information from its observation and the inference to produce the action. The authors conducted experiments on SMAC and GRF to assess the effect of MA$^2$E.", "questions": "1. The official code of COLA is implemented based on the original *pymarl*, while MA$^2$E is implemented based on *pymarl2*. Is there any issue of fairness?\n2. How does the training cost (measured in GPU hours) of MA$^2$E compare to that of other methods?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is easy to follow. The motivation is clear, and the presentation is of good quality.\n2. The authors conducted sufficient experiments, which provide a comprehensive demonstration of MA$^2$E. \n3. The proposed method can be applied to various multi-agent reinforcement learning algorithms, including value-based and policy-based.", "weaknesses": "1. The novelty of this paper is relatively limited. It simply integrates a masked autoencoder into the decision-making process of agents, which is more oriented to engineering and makes little theoretical contribution. It\n2. Some ablation studies are conducted on only a single scenario (e.g., 3s_vs_5z). The results may be contingent.\n3. The selected baselines are not strong enough. Many MARL algorithms also modify the decision-making process of agents while keeping the constraints in observability and communication, such as RODE [1], LDSA [2], and SORA [3]. The authors should compare MA$^2$E with these methods to make the results more convincing.\n4. Compared to the baselines, the training of MA$^2$E is more complex and computationally expensive, but the performance is not significant enough. In other words, this method is not cost-effective.\n\n[1] Wang, T.; Gupta, T.; Mahajan, A.; Peng, B.; Whiteson, S.; and Zhang, C. 2020c. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523.\n[2] Yang, M.; Zhao, J.; Hu, X.; Zhou, W.; Zhu, J.; and Li, H. 2022. Ldsa: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35: 1698–1710.\n[3] Zhou G, Xu Z, Zhang Z, et al. SORA: Improving Multi-agent Cooperation with a Soft Role Assignment Mechanism. International Conference on Neural Information Processing. Singapore: Springer Nature Singapore, 2023: 319-331.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)