Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Authors: Andreas Kontogiannis
Konstantinos Papathanasiou
Yi Shen
Giorgos Stamou
Michael M. Zavlanos
George Vouros
Venue: ICML 2025 poster (2025)
Keywords: Cooperative Multi-Agent Reinforcement Learning
State Modelling
Sparse Reward
Primary area: reinforcement_learning->multiagent

Abstract: Learning to cooperate in distributed partially observable environments with no communication abilities poses significant challenges for multi-agent deep reinforcement learning (MARL). This paper addresses key concerns in this domain, focusing on inferring state representations from individual agent observations and leveraging these representations to enhance agents' exploration and collaborative task execution policies. To this end, we propose a novel state modelling framework for cooperative MARL, where agents infer meaningful belief representations of the non-observable state, with respect to optimizing their own policies, while filtering redundant and less informative joint state information. Building upon this framework, we propose the MARL SMPE$^2$ algorithm. In SMPE$^2$, agents enhance their own policy's discriminative abilities under partial observability, explicitly by incorporating their beliefs into the policy network, and implicitly by adopting an adversarial type of exploration policies which encourages agents to discover novel, high-value states while improving the discriminative abilities of others. Experimentally, we show that SMPE$^2$ outperforms a plethora of state-of-the-art MARL algorithms in complex fully cooperative tasks from the MPE, LBF, and RWARE benchmarks.

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Andreas Kontogiannis 12 Konstantinos Papathanasiou *3 Yi Shen
Giorgos Stamou'
Michael M. Zavlanos" George Vouros
Abstract
learning agents in order to maximize their shared utility func- tion. Many real-world applications can be modeled as fully cooperative MARL problems, including multi-robot coop- eration (Alami et al., 1998), wireless network optimization (Lin al., 2019). self-driving cars (Valiente al., 2022), air traffic management (Kontogiannis & Vouros, 2023), and search & rescue (Rahman et al., 2022).
Learning to cooperate distributed partially ob- servable environments with no communication abilities poses significant challenges for multi- agent deep reinforcement learning (MARL). This paper addresses key concerns in this domain, fo- cusing on inferring state representations from in- dividual agent observations and leveraging these representations to enhance agents' exploration and collaborative task execution policies. To this end, we propose novel state modelling frame- work for cooperative MARL, where agents in- fer meaningful belief representations of the non- observable state, with respect to optimizing their own policies, while filtering redundant and less in- formative joint state information. Building upon this framework, we propose the MARL. SMPE? al- gorithm. In SMPE", agents enhance their own pol- icy's 's discriminative abilities under partial observ- ability, explicitly by incorporating their beliefs into the policy network, and implicitly by adopt- ing an adversarial type of exploration policies which encourages agents to discover novel, high- value states while improving the discriminative abilities others. Experimentally, we show that SMPE2 outperforms state-of-the-art MARL algo- rithms in complex fully cooperative tasks from the MPE, LBF, and RWARE benchmarks.
In most world applications, agents, acting in a decentral- ized setting, have access to the environment state through mere observations. However, collaborating under partial observability is a challenging problem, inherent from the non-stationary nature of the multi-agent systems. To address this problem, different approaches have been proposed: e.g.- approaches which aim agents to model other agents (e.g.. in (Hernandez-LLen Leal et al., 2019)), to communicate effectively with others (e.g., in (Guan et al., 2022b)), or to leverage advanced replay memories (e.g., in (Yang et al., 2022)).
This paper draws motivation from fully cooperative MARL real-world applications (Papadopoulos al., 2025) and fo- cuses the celebrated CTDE schema (Lowe et al., 201 7a), where agents aim to learn effective collaborative policies. During training, agents share information about their ob- servations, but in execution time they must solve the task given only their own local information. Specifically, we are interested in settings where agents lack explicit com- munication channels during execution. Such settings of particular interest, because, while communication-based methods leverage inexpensive simulators for training, they may incur substantial computational overhead when exe- cuted in real-world environments (Zhu et al., 2022; Zhang & Lesser, 2013). This paper aims agents to infer meaningful beliefs about the unobserved state and leverage them for enhanced cooperation with others.
1. Introduction
Background and Motivation. In cooperative multi-agent deep reinforcement learning (MARL), the goal is to train the
"qqual contribution 'School of Electrical and Computer Engi- neering, NTUA, Greece 'Archimedes, Athena Research Center, Greece Treece'Departtneee Deppatmennt of Mathematics. ETH Zurich 'Dept. of Me- chanical Engineering & Materials Science, Duke University Dept. of Digital Systems, University Piracus, Greece. Correspondence to: Andreas Kontogiannis <andrcaskontogiannise mail. .ntillntuarrrr
Agent modelling (AM) (Albrecht & Stone, 2017), also re- ferred to opponent modelling (He et al., 2016; Foerster et al., 201 8a), has been proposed as solution to infer be- liefs about the global state, or the joint policy, under partial observability. Applications of AM in MARL have been extensively explored (Papoudakis et al., 2021; Raileanu et al., 2018: Sun et al., 2024; Hernandez-Leal Nguyen al., 2023; Tan & Chen, 2023; Gupta & Kahou,
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
al., 2019;

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
2023; Fu al., 2022), aiming agents to infer belief repre- sentations about other agents' policies the unobserved state. Nonetheless, current AM approaches may pose chal- lenges in enhancing agents' policies for the following rea- sons: (a) Standard AM may infer belief representations that are suboptimal for enhancing the agents' policies, because these representations are not learnt w.r.t. maximizing the agent's 's value function (e.g., in (Papoudakis et al., 2021: Pa- poudakis & Albrecht, 2020; Nguyen et al., 2023; Hernandez- Leal et al., 2019; Fu et al., 2022)). (b) Standard AM does not account for redundant, less informative, joint state fea- tures in learning the agent's ' belief' (e.g., in (Railcanu et al., 2018; Hernandez-Leal et al., 2019; Papoudakis et al., 2021; Nguyen et al., 2023; Tan & Chen, 2023)), which has been shown to harm performance (Guan et al., 2022b). (c) AM representations are not leveraged in MARL to improve the agents' initial random exploration policies, thus being inef- fective for enhancing policies in sparse-reward settings (as we show in Section 4.4). Moreover, AM approaches may be impractical because they typically involve a single learnable controller (c.g., in (Papoudakis et al., 2021; Nguyen et al., 2023; Fu et al., 2022)), or make strong assumptions regard- ing: the observation feature space; e.g., assuming a priori knowledge about what the observation features represent (Nguyen et al., 2023; Tan & Chen, 2023)), the nature of the game (e.g., being applicable only to Gam--ame settings (Sun et al., 2024)), or settings where agents observe other agents' actions (Hu & Foerster, 2019; Hu et al., 2021; Fu et al., 2022) in execution time.
the state inference capabilities of other agents and ultimately facilitating better cooperation.
Main Contributions. (a) We propose a novel state mod- elling framework for cooperative MARL under partial observability. In this framework, agents are trained to infer meaningful state belief representations of the non- observable joint state w.r.t. optimizing their own policies. The framework assumes that the joint state information can be redundant and needs to be appropriately filtered in order to be informative to agents. Also, the framework entails multi-agent learning and does not impose assumptions on either what the observation features represent, or access to other agents' information during execution. (b) Building upon the state modelling framework, we propose "State Modelling for Policy Enhancement through Exploration"! (SMPE*), a MARL method which aims to enhance agents' individual policies' discriminative abilities under partial ob- servability. This is done explicitly by incorporating their beliefs into the policy networks, and implicitly by adopting adversarial exploration policies aiming to discover novel, high-value states while improving the discriminative abil- ities of others. More specifically, SMPE? leverages (i) variational inference for inferring meaningful state beliefs, combined with self-supervised learning for filtering non- informative joint state information, and (ii) intrinsic rewards to encourage adversarial exploration. (c) Experimentallyy we show that SMPE2 significantly outperforms state-of-the- art (SOTA) MARL algorithms in complex fully coopera- tive tasks of the Multiagent Particle Environment (MPE) (Mordatch & Abbeel, 2018; Lowe et al., 2017b). Level- Based Foraging (LBF) (Albrecht & Stone, 2017) and the Multi-Robot Warehouse (RWARE) (Papoudakis et al., 2020) benchmarks, several tasks of which have been highlighted as open challenges by prior work (Papadopoulos et al., 2025).
Taking the above significant challenges into account, this paper addresses the following research questions:
Q1: Can agents learn to infer informative (latent) state representations given their own observations, in order to enhance their own policies towards coordination?
Q2: Can agents leverage the inferved state representations to efficiently explore the state space and discover even better policies?
2. Preliminaries
2.1. Cooperative MARL as a Dec-POMDP
A Dec-POMDP (Olichock et al., 2016) for an N-agent cooperative task is tuple (S, A, F,,,F, where S is the state space, A is the joint action space A A1 x ... x AN, where A₁ is the action space of agent i, P(s' s,a) S x A + [0, 1] is the state transition function, r(s,a) : 5 x A + R is the reward function and 7 E [0, 1) the discount factor. Assuming partial observ- ability, each agent at time step does not have access to the full state, yet it samples observations of E O, accord- ing to the observation function F((s) S - O₂- Agents' joint observations are denoted by E O and are sampled according to F = II. F. The action-observation history
High-level Intuition. To understand the high-level intu- ition guiding our approach, imagine fully cooperative task from the well-known LBF benchmark, where agents must first engage in extensive joint exploration to identify specific food target. Once identified, all agents must then execute coordinated joint actions, simultaneously consum- ing the target. At each time step, each agent-baseed on its current local information-attemn informative state information (e.g., the food target's location and the positions of other agents with whom it must cooper- ate). The agent then refines its policy using informative state inferences accumulated throughout the entire trajectory. To enhance joins exploration, each agent is further incentivized to implicitly discover novel observations, thereby improving
to infer unobserved yet
'Our official source code hutp://gthhb.ccm/ddacdluummg
be found
2

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Contraized Critc
(,,000 (.t,5))
add intrinsic reward
(4,01000,
Miattlc
advantages
Actor
self- holf-supervised
adversarial exploration
fill observability
Filters
partial COSErVaSELLty
The (oi Mar
Gradient flom
Figure 1. Overview SMPE": SMPE2 is built on top the MAA2C algorithm (Papoudakis et al., 2020). Each agent's ' actor (blue hox) has partial observability. Conditioned on agent' own observation o², it reconstructs other agents' observations o using variational encoder-decode (ED) which infers a belicf state embedding z'. To filter non-infoon informaatii joint state information and learn only from informative features, the actor utilizes agent modelling (AM) filters (w") which filter the reconstruction targets of ED in self-supervised learning manner. Moreover, the embeddings z' are exploited to provide intrinsic rewards f', order to facilitate the proposed adversarial exploration schema. Finally, the actor exploits the belief embeddings z' enhance the policy (a) M4, =[). Concurrently, SMPE leverages two critic models (red box), namely (s(s) and ($) (parameterized by & and k accordingly), both accessing full observability. VE is the standard critic of backbone MAA2C and (&) approximates explicitly the joint value function at modified states purified by the AM filters w'. During execution agents do not have access to others' ohservations, but use the inferred x² to discriminate effectively between different states despite partial observability. The algorithm SMPE? can be found Algorithm (Appendix).
for agent at time denoted by hy E Hi, which in- cludes catiionobbervation pairs until t-1 and of. on which the agent can condition its individual stochastic policy Tb. (a) ht) H; x A; [0, 1], parameterised by 0,, The joint action of all agents other than is denoted by a and we similar convention for the policies, i.e., mg
State modelling assumes a belief space Z, of latent vari- ables z², such that z Pw, (- o²) contains meaningful information about the unobserved state, informative for opti- mizing the agent's states contain information redundant for optimising agents' individual policies. Specifically, we consider a joint state feature to non-informative for agent if:
policy. In particular, assume that
The joint policy is denoted by Te. with parameters 0 E θ
The objective is to find an optimal joint policy which satisfies the optimal value function V maxg Bu-rop./
it irrelevant to agent for maximizing its future rewards (if assuming that the agent had access to it)
EE r/ri.
it cannot be inferred through z², in the sense that the agent cannot predict it conditioned on its own observa- tion due to partial observability and con-stationarity.
2.2. The state modelling optimization framework
The main idea behind the proposed state modelling (SM) optimization framework to allow agents, conditioned only on their own observations, to infer meaningful beliefs of the unobserved state which would help them optimize their individual policies. Our state modelling framework connects statetrepresentatioon learning with policy optimization.
Considering the above assumptions, the state modelling objective is find meaningful latent beliefs of with respect to optimizing the joint policy gg. resulting from individual policies Te, parameterized by vh, i.e. nw (at R 2f)
H, x Z, x A; + [0, 1]. as follows:
Formally. we consider that each agent i aims to learn model of the non-observable joint state: The model infers a probability distribution over beliefs regarding the non- observable joint state, based on agents' local observations. The model is parameterized by w; and denoted by Pw,.
VšM max max Vsar(w,w)
max Eerop Ea
Σ
3

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
where VSM is the optimal value function under SM.
represents the importance of each of other agents' informa- tion to agent i's state modelling.
Proposition 2.1. The state modelling objective equals the DecPOMDP objective; i.e. Vša= =
Formally, for each agent i, SMPE? considers a modification of the state modelling framework of (1) to the following maximization problem:
Although both objectives are intractable. Proposition 2.1 implies that VsM allows us to explore how agents can form meaningful beliefs 2' about the unobserved states with suffi- cient stateddisrimminating abilities to enhance their own poli- cies without constraining the candidate policies, i.e. those that utilize 2², to be suboptimal w.r.t. the value function.
maximize Wi.P
VSM (w, +)) +A &n:o,, w' (2)
where (in w" o ") denotes element- wise multiplication and we define ELBO(w,, (wi, Øz ; ; , w" '0
1) to be equal to
3. State Modelling for Policy Enhancement through Exploration (SMPE?
E Eg'nge [log Po. (w' o {)] KL (Awi o') (l()))
that is, the Evidence Lower bound (Kingma & Welling, 2013) which uses o-i filtered by w" as the reconstruction targets.
We propose state modelling MARL method, called SMPE? The method can be separated into two distinct, but concurrent, parts. The first part (Section 3.1) involves self-supervised state modelling, where each agent learns a model to infer meaningful belief state representations based on own and other agents' observations. The second part (Section 3.2) involves the explicit use of the inferred repre- sentations encourage agents towards an adversarial type of multi-agent exploration through intrinsic motivation. The learning process of SMPE? is illustrated in Figure 1.
Although problem (2) is also intractable, the ELBO term would allow SMPE2 to find good solutions: ED aims to identify informative observation features in the reconstruc- tion, through the AM filters, leading to 2 which enhance the individual agents' policies towards cooperation. do- ing so, SMPE? entails enhanced performance of the joint policy by finding state beliefs 2¹ w.r.f maximizing the value function. This comes in contrast to other approaches (e.g.. (Papoudakis et al.. 2021; Papoudakis & Albrecht, 2020; Nguyen et al., 2023; Hernandez- et al., 2019; Fu et al., 2022)) which may suffer from distribution mismatch (Ma et al.), as 2 is disconnected from the policy. We note that X in (2) a hyperparameter ensuring that the ELBO term does not dominate the state modelling objective VsM.
3.1. Self-supervised state modelling
To learn a representation of the unobserved state, aim agents to learn the reconstruction of informative features of other agents' observations using only their own observa- tions. The reconstruction aims at inferring latent beliefs, 2², about the unobserved state, assuming that joint observations provide sufficient evidence to discriminate between states. SMPE uses amortized variational inference (Kingma & Welling, 2013), i.e., the reconstruction model of agent is probabilistic encoder (g... :decoder (Po.) (ED). One would assume that ideally ED should perfectly predict (ō-²) other agents' observations (o so that z' could be as informative about what other agents observe as possible. However, as it was highlighted in prior work (Guan et al., 2022b), using the full state information an extra input the policy, even when utilizing compressed embedding, may harm perfor- mance due to redundant state information non-informative to agent i.
Based on (2), in the reconstruction part of SMPE?, ED has as prediction target the other agents' observations, or fil- tered by w². More specifically, the ED per agent i minimizes the following self- superviee reconstruction loss (using the reparameterization trick (Kingma & Welling, 2013), with wi being Gaussian):
Lyce (wi, 6s o²")) | -0 o
01
(3)
where we use targets w² to filter the target observations o- in order to stabilize the training of w².
Both (wi, Φ.) and w² are learned in self-supervised manner, since the targets in Equation (3) are explicitly factorized by the AM filters. Note the importance of the AM filters w³: (a) With it, although the target of ED grows linearly with the number of agents, only features that can be inferred through 2f remain as part of other agents' observations in the reconstruction loss. (b) Without it, it would be challenging to infer meaningful embeddings 2", due to non-informative joint state information. Thus, w' explicitly controls the reconstruction of the initial other agents' observations, since it adjusts how much ED would penalize the loss for each target feature.
To mitigate the use of redundant state information, SMPE? filters out non-informative features of o o-,, i.e., state fea- tures that are irrelevant to agent for maximizing its future rewards and cannot be inferred through 21. To achieve this, we introduce agent modeling (AM) filters, denoted by w which serve as learnable weight parameters one for each of the other agents j E -i. SMPE' utilizes w/= (6)) (o)),, where o is parameterized by an MLP, and a is the sigmoid activation function, ensuring that the filter values satisfy ur E [0. 1]. Intuitively, w has an AM interpretation, as it
4

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
However, if both w² and w6 are equal to O, then these assign- ments are solutions to (3). To ensure that w² does not vanish to zero throughout all dimensions, we add the following regularization loss:
for both individual and collective benefit. Specifically, our exploration strategy encourages each agent to effectively explore its own belief space so that it discovers novel z'. To do so, given that 2' is solely conditioned on o', the agent is implicitly motivated to discover novel observations that must lead to novel z². Crucially, our exploration schema implies an adversarial exploration framework fostering co- operation: Agent
L Laorm (.; = -1
(4)
To make the embeddings 2 variational (and thus probabilis- tic as in the definition of state modelling), following the problem in (2), we add the standard KL divergence regular- ization term, as follows:
intrinsically motivated to discover novel of (which lead to novel 2²) which at the same time constitute unseen ED prediction targets for the others' re- construction training. Therefore, these targets aim to adver- sarially increase the losses of other agents' reconstruction models. By doing do, agent i, except only for searching for novel o², implicitly strives to assign adversarial targets to other agents -i, to help them form better beliefss about the global state. In the same way, since 2 is informative of both of and oー agent i, leveraging o- is also urged to form well-explored belief about what others observe, enhancing cooperation.
LxI. (w:)) = (1, (=*)o") II)())),
- N(0,
(5) As for the policy optimization part, SMPE? can be imple- mented using any actor-critic MARL algorithm. We imple- ment SMPE2 using MAA2C (see Appendix C. due to its good performance on various benchmarks (Papoudakis et al., 2020; Papadopoulos et al., 2025) and its efficient natural on-policy learning. Following the definition of the state modelling problem, we aim to ensure that w² (and thus z²) incorporate information relevant to maximizing V and thus, w² to be capable of filtering non- con-informative state fea- tures irrelevant maximizing agent's future rewards. To do so, alongside MAA2C's standard critic (parameterized by {), we train an additional critic (parameterized by k)) which exploits w'. Thus, we minimize the following losses:
To leverage the above benefits, we adopt a simple count- based intrinsic reward schema based on the SimHash algo- rithm, similar (Tang et al., 2017), but instead of hashing the agent's 's observations, utilize the SimHash function SH Z - 2 and calculate the intrinsic reward f as follows: f = 1/ / (Vn(S((H2))) where n (="H(=))) represents the counts of SH (=*) for agent i. Agent uses the modified reward F = (ri + Bf%), where hyperparameter / controls the contribution of the intrinsic reward.
hash 2² Z. Specifically, we
Leiitic (E) ri + V (s(+1) - Vi" (sx)
(6)
eritii (0, k) [ré + (8++1) - V/5 ($:)]
The choice of SH is due the fact that it allows nearby 2² to be transformed into nearby hash values at low computational cost. A key note here is that the domain of the SH function is dynamic. To make intrinsic rewards more stable and ensure the assumption of i.i.d. training data, 2' becomes more stable by avoiding large changes in the ED parameters between successive training episodes. To do so, similarly to (Papoudakis et al., 2021), we perform periodic hard updates of W; and de using a fixed number of training steps as an update period.
where و = of @ (w" is the filtered state from the view of agent i's modelling, with @ meaning vector concatenation, and superseript for target network. Overall, the loss for learning the state belief representations for agent is:
Lencodings = Leitic + Asee Lioss + Anomn Lnorn + AKL. LKL (7)
where Arees Anomm and ÅKL are hyperparameters weighting the corresponding loss terms. The actor of each agent uses policy π = o. enhanced by 21. Thus, the actor loss is:
Remark 3.1. For the interested reader, we demonstrate in Appendix E.4.7 that the intrinsic rewards do not fluctuate; instead, they decrease smoothly throughout training until they reach minimal plateau, as expected.
Tracuor (wi) = -3 H
( (Ir (at Ni z{))
- log Tv.(a) (a)1 h4, zt) (1) + VV
Therefore, the overall SMPE loss is as follows:
Why is ED conditioned only on of? We choose ED to be conditioned solely on of, instead of h². In particular, since z leveraged for intrinsic exploration, we make 2f solely conditioned on o², so that the agent will be intrinsically re- warded more if the current observation was novel and led to novel z². On the other hand, if ED was indeed condi- tioned on h²' then the intrinsic exploration would be less meaningful, since the main goal now would be to discover novel trajectories instead of novel observations (that lead to
LSMPE= -Irccur + Legitic + Lencondings
3.2. A Adversarial Count-based Intrinsic Exploration
We further empower the agents' policies to reach novel, hagh-value states, even in sparse-reward settings. To do so, we harness the rich state information captured by the be- lief 2f and design intrinsic rewards which naturally account
5

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
novel z"), thus making the novelty of the current observation excessively compressed within z¹. For instance, if the agent discovered a novel, high-value observation (thus contained in high-value global state) within well-explored trajec- tory, then the intrinsic reward would be quite low (because the whole trajectory would not be novel) and thus possibly unable to help the agent find a better policy. The above conceptual motivation does not defeat the assumption of partial observability, because the policy network is indeed conditioned on hy and therefore implicitly leverages a be- lief representation conditioned on all time steps, that is all per-time-step beliefs zi.
4.1. Results on the MPE benchmark
In this section, we evaluate the proposed method without intrinsic rewards for exploration (SMPE) two fully co- operative MPE tasks: Spread and Double Speaker-Listenee To assess scalability, we examine four scenarios with 3, 4, 5, and 8 agents in the Spread task, following a setup similar to (Ruan et al., 2022). Due to space limitations, results for Double Speaker- Speaker-Listener Listener are provided in Appendix B. As depicted in Figure 2, SMPE demonstrates superior per- formance in Spread, showing significant improvements in overall performance. Notably, as the number of agents and landmarks increases, SMPE consistently outperforms other methods by enabling agents to form informative beliefs about the unobserved state. This allows agents to maintain awareness of available landmarks even with increased com- plexity, a capability not easily achieved by other methods.
4. Experimental Setup
In our experimental setup, we evaluate MARL algorithms on three widely recognized benchmarks: MPE (Mordatch & Abbeel, 2018; Lowe et al., 2017b), LBF (Albrecht & Stone, 2017), and RWARE (Papoudakis et al., 2020). Due to space constraints, technical details for these benchmarks can be found in Appendices E.I, E.2, and E.3, respectively. These benchmarks often require substantial coordination and ex- ploration capabilities in order to discover effective joint policies and have been used evaluate MARL algorithms in many related works, including (Papadopoulos et al., 2025; Christianos et al., 2021; Papoudakis et al., 2021; Guan et al., 2022a; Yang et al., 2022; Papoudakis et al., 2020).
4.2. Results on the LBF benchmark
In this section, we evaluate the full SMPE2 on six fully cooperative LBF tasks. As Figure 3 shows, SMPE2 achieves superior performance compared to the other SOTA methods. In contrast to the other methods, SMPE manages to fully solve challenging sparse-reward LBF tasks, including 2s- 9x9-3p-2f and 4s-1/x/1-3p-2f both of which have been highlighted as open challenges by prior work (Papadopoulos et al., 2025). As for the other LBF tasks, SMPE either converges faster (a) to an optimal policy, or (b) to a good policy, or constantly achieves better episodic reward over time, even with large grid (see 7s- 7s-20x20-5p-311
Regarding the questions stated in the introduction, to address Q1, we first verify the effectiveness of the inferred state modelling embeddings of the proposed method on fully cooperative dense-reward tasks of the MPE benchmark. To this aim, in MPE we evaluate our method without using the proposed intrinsic rewarding schema. To answer Q2, we evaluate the effectiveness of full SMPE? on complex, fully cooperative, sparse-reward tasks of the LBF and RWARE benchmarks. In MPE we compare our method with MAA2C, COMA (Foerster et al., 2018b), MAPPO (Yu et al., 2022) and the wransformer-basee ATM (Yang et al., 2022). In LBF and RWARE, we also include the SOTA intrinsic motivation based methods: EOI (Jiang& & Lu, 2021), EMC (Zheng et al., 2021) and MASER (Jeon et al., 2022), as they cover a wide range of different approaches for exploration in MARL; namely through diversity, curiosity and sub-goal generation respectively.
SMPER's 's success stems from its alignment with LBF's need for precise coordination among agents (through accurate beliefs about the unobserved state and good exploration policies) to collect all foods optimally. On the other hand, EMC and MASER completely fail to improve the well- known poor performance of QMIX (Rashid et al., 2020) in LBF (e.g., see (Papadopoulos et al., 2025; Papoudakis et al., 2020)), because both methods highly rely on the initial random policies and, as a result, they generate misleading intrinsic rewards based on low-value episodic data or irrele- vant sub-goals, respectively. The only exception on this is the 13-12x/2-2p-- task, only because this task is the least sparse and finding a good, non-optimal policy is not that hard. EOI, which is built on top of MAA2C, does not per- form well as MAA2C, because it encourages the agents to explore through diversity, which may be unnecessary to the homogeneous agents in LBF.
Results are averaged over six random seeds, and the primary metric is the unnormalized average episodic reward. We report 25-75% confidence intervals (as in (Zheng et al., 2021)). Following (Papoudakis et al., 2020), we set the total number of steps to 10M for MPE and LBF, and 40M for RWARE.
4.3. Results on the RWARE benchmark
We evaluate SMPE? on three hard, fully cooperative RWARE tasks; namely tiny-2ag-hard, tiny-4ag-hard and small- mull--4a-hhrde hard. As shown in Figure 4, our SMPE? achieves
6

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Spread-3
spread-4
Spread-5
Spread-8
-000
Egioodic Reward
-160
-300-
Ejisodic neward
Eplcodic Reward
-3500
-1280-
-8300
Episodic
-500
-2003
8900
2203
-2003
-4400
-000
1.0
14
-2003
0-4
1.0
-2500
0.0
0.5
1.0
Steps
327
SMPE
Sleps
MAAZC
MAPPO
ATM
Stepss
COMA
30
Steps
Figure 2. Results on the MPE benchmark
2f coop
3p coop
2p 2f coop
Epoodt: Rewerd
0.0
Fpwodtle Bawward
0.8
Egcoodicc Riswardd
0--
0.4
2.4
0.0
0.0
0/0
0.0
0.4
0.0
0.4
0.5
0.2
0.4
Steps
Steps
Steps
Ibf-7s.20220-p- 34-coop
lof-25-11xD] 3p-2f-copp
Ibf-45-1 11l1 1-w.111------0 3p-2f-copp
0.8
Egtedic Reward
Fplodde Besuard
0.c
Eptcadi Roward
3.8
0.-
62-
0.0
0.4
0.8
SMPE?
MAA2C
MAPPO
ATM
E01
MASER
EMC
COMA
Figure 3. Results on the LBF benchmark
superior performance compared to the SOTA methods across all three tasks. These RWARE tasks are particularly challenging because they require: (1) effective exploration strategies deal with sparse rewards; that only very spe- cific sequences of actions result to positive rewards (first load specific shelf and then unload into empty shelf), (2) agents to coordinate so that they avoid monopolizing all tasks individually or adopting policies that obstruct others, particularly in narrow passages, and (3) accurate modelling of the state due to excessive partial observability.
the adversarial exploration schema?
Q4: How expressive is the state modelling embedding zt?
Q5: Does SMPE2 outperform other modelling methods?
Q6: Is SMPE flexible? Can it be combined with other
MARL backbone algorithms? (see Appendix E.4.9)
Q7: Do we really need second critic for training w" w.r.t. policy optimization? (See Appendix E,4.3)
First, we address question Q3. Figure validates the com- ponent selection in SMPE? It depicts (up) the importance of the AM filters (w²) in better episodic reward and con- vergence to good policies and also the superiority of the proposed adversarial exploration schema over the standard method (Tang et al., 2017). Here, we note that the complete failure of SMPE (no intr) is attributed to the fact that al- though agents are empowered by state modelling abilities, may be difficult to adopt good exploration policies and thus to reach high-value states. Figure (up) highlights that both excessive cooperation and good exploration to achieve good performance, as the baseline SMPE meth- ods do not perform well. Thus, in these tasks, we would not have good exploration without good state modelling for effective collaboration, neither good collaboration without good exploration. Moreover, Figure 5 (down) demonstrates
The failure of MASER and EMC is attributed to the first of the reasons. EOI displays good performance because it encourages the agents to be more diverse, thus some of them are able to find good policies. However, due to the second reason it completely fails in small-4ag-hard because of insufficient coordination among agents. In contrast, SMPE? outperforms the SOTA methods, as: It addresses the third reason through modelling explicitly the joint state and also the first and second reasons through the effectiveness of the adversarial exploration schema.
required
4.4. Ablation Study
In our ablation study, address the following questions:
Q3: How important state modelling, the AM filters and

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
ivare-tin-2ag-hhard
rware-tiny-ta-haard
rware-smail-fag-hard
20.6
10
10.0
1.2
13
1.0 MAPPO
SMPEA:
MAA2C
ATM
ED1
.MASER
EMC
100MA
Figure Results on the RWARE benchmark
1bf-2s-11x11-pp22666
1bf44s-11x11-3p-2--oop
Eplcodic Reward
8.8
Eplcodic
20872
SMPC und Inl)
50mte (ro Mllanal
3001 Iolo, 1990
396 100
haad
hware-tiny-aag-hard
201-
Episodic Reward
Episodic Reward
02
55
28
13
28 Steps
25
37
6.
355
2.5 Steps
23
39
33
SMPE
SMME! [no, cribic
Figure Ablation study: (up) Validating component selection: SMPE without intrinsic rewards (no intr), AM filters (no filters), LKa. (no kf), Tenom (no 1. norm), and with standard SimHash (Tang et al., 2017) replacing our exploration schema (obs rew). (down) Learning w (and thus z²) w.r.t. policy optimization: we use SMPE? without Liiic (no_critic_w) as baseline.
the impact of learning state representations w.r.t. policy op- timization (as proposed in (1)), which can yield significant improvements in episodic reward and policy convergence.
LBF tasks (benchmarks where LIAM was also originally tested). SMPE? significantly outperforms the baselines due to the following reasons: (a) MIJAM's objective is exac- erbated by modelling the actions of others, leading to an inference problem with an exponentially large number of candidate target actions, (b) SIDE does not use the inferred z in execution, (c) both methods do not account for non- informative state information, and (d) both methods do not leverage the inferred representations to improve joint explo- ration.
To address Q4, in Figure we present t-SNE representa- tions (commonly used in MARL as in (Papoudakis al., 2021; Xu et al., 2023; Liu et al., 2024)) of the state mod- elling beliefs of of the three agents in LBF 25- 1x111-p-22 task. The figure demonstrates that when LKL. is enabled, agents form cohesive beliefs: Their beliefs coincide in re- gions covering large area, while maintaining individual beliefs in regions covering smaller area. Conversely, with- TAL., agents' beliefs show less coherence, resulting in poorer joint exploration, as depicted in Figure 5. Table (Appendix E.4.8) further validates these findings.
MARL and Attention Modules. MARL approaches (e.g.. (Sunehag al., 2018; Rashid et al., 2020; Son al., 2019)) represent the joint state-action value vidual models learnt based on the global reward. Atten- tion modules (Vaswani et al., 2017) have been proposed: to encourage the cooperation of agents via agent-centric mechanisms (Shang et al., 2021), to estimate the value func- tions for explorative interaction among the agents (Ma et al., 2021), advanced replay memories (Yang et al., 2022).
function of indi-
To address question Q5, we compare SMPE? to SIDE (Xu et al., 2022) and a multi-agent extension of the popular AM method LIAM (Papoudakis et al., 2021) (denoted by MLIAM). More details about MLIAM can be found Ap- pendix C.2. Figure 6 illustrates the comparison in MPE and
8

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Spread44
Sprraad-5
Ibf-2s 1B-22-9x9-3p22..c00
D4-6--1x11-3p22-0000
1.3
5.0odic Reward
Edtoudic Reward
Eoigadic Reward
-3000-
1200
-1406
Steps
107
SMPE
MILAM
SIDE
Steps
107
Steps
167
Figure 6. SMPE2 against custom implementation of the multi-agent extension of LIAM (Papoudakis et al., 2021) and SIDE (Xu al., 2022)
SNE Visualization atz', Ibt 1p 27, timestopp 45
t: VISUaI zatiozzatinnnnn ot IbF 25 11-111111:3pp22 2t, timestep: 45
E-SNE Dimmenston
E:SNE Dimension
Dimension
ü-SNE Dimension
Figure 7. t-SNE visualization of the state modelling embedding 2' (for the three agents; agent I: red, agent 2: green, agent 3: blue) in an LBF task at the 45th time step; (left): SMPE?, (right): SMPE (no (no for ablation study with LgL 0.
for efficient credit assignment (Zhao et al., 2023b), and for learning good policies (Ma et al., 2023; Wen et al., 2022: Zhao et al., 2023a).
(Yuan et al., 2022; Hu & Foerster, 2019; Hu et al., 2021; Fu et al., 2022), or considers only team-game settings (Sun et al., 2024). Additionally, approaches such (Raileanu et al., 2018; Hermandez-Leal et al., 2019; Nguyen al., 2023; Tan & Chen, 2023; Xu et al., 2022) do not consider non-informative joint state information in learning beliefs, which can detrimentally affect performance (Guan et al., 2022b).
Agent Modelling (AM). AM (Albrecht & Stone, 2017) has been proposed to model the policies of other agents or the unobserved state. Bayesian inference has been used to learn representations of probabilistic beliefs (Zintgraf et al., 2021; Moreno et al., 2021; Zhai et al., 2023), or even for explic- itly modelling concepts derived from the theory of mind (Nguyen al., 2020; Moreno et al., 2021; Hu et al., 2021). LIAM (Papoudakis et al., 2021) uses an ED to learn em- beddings, by reconstructing the local information of other agents, and add them as extra input for policy learning. key distinction between standard AM (Hernandez-Lon et al., 2019; Papoudakis et al., 2021; Nguyen et al., 2023) and our state modelling lies in their utilization: the former serves as auxiliary task disconnected from policy opti- mization, while the latter does not necessitate optimal full reconstruction, but only good reconstruction of the filtered targets to find good policies. Also, AM typically involves a single controller interacting with non-learnable agents (Hernandez-LLes) et al., 2019; Papoudakis Albrecht, 2020; Papoudakis et al., 2021; Nguyen et al., 2023; Fu et al., 2022), or makes strong assumptions regarding prior knowledge of observation features (Tan& & Chen, 2023; Nguyen et al., 2023), access to other agents' information during execution
5. Conclusion and Future Directions
In this paper, we studied the problem of inferring infor- mative state representations under partial observability and using them for better exploration in cooperative MARL. We proposed a state modelling optimization framework, on top of which we proposed a novel MARI. method, namely SMPE", Experimentally, we demonstrated that SMPE out- performs state-of-te oftte-of-the-aane -he- methods in complex MPE, LBF and RWARE tasks.
In future work, we are interested in the following open challenges: (a) Can we further improve state modelling by incorporating transformers into the architecture? (b) How can we use state modelling to improve scalability in MARL? (c) Does state modelling adapt well in stochastic settings (i.e. settings with noisy observations, or more complicated dynamics)?

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Acknowledgements
with selective parameter sharing. In International Con- ference Machine Learning, pp. 1989-1998. PMLR, 2021.
This research was supported in part by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NexIGenerationEUU Program.
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning with opponent- learning awareness. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pp. 122- 130, 2018a.
Impact Statement
Multi-Agent Deep Reinforcement Learning (MARL) holds promise for wide array of applications spanning robotic warehouses, search&rescue, autonomous vehicles, software agents, and video games, among others. Partial observ- ability in these settings is an inherent feature much decentralization: This imposes challenges for the coordi- nation and cooperation of agents that this work addresses. Indeed, this paper contributes the state modelling frame- work and SMPE algorithm to the advancement of MARL models for such applications, in large and continuous state- action spaces, close to real-world problems. However, it is essential to acknowledge that the practical implementation of MARL methods faces significant challenges, including issues of limited theoretical guarantees, poor generalization to unseen tasks, explainability, legal and ethical consider- ations. These challenges, although beyond the immediate scope of our work, underscore the necessity for prioritizing extensive research in MARL. The overarching objective is to develop MARL agents capable of operating safely and addressing real-world problems effectively. It is imperative that many MARL methods undergo rigorous testing before deployment. While the potential benefits of safe, accurate, and cost-effective MARL applications are substantial, in- cluding the reduction of human effort in demanding tasks and the enhancement of safety, achieving these outcomes requires meticulous validation and refinement of the under- lying methodologies.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradi- ents. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018b.
Fu, H., Tian, Y., Yu, H., Liu, W., Wu, S., Xiong. J., Wen, Y., Li, K., Xing, J., Fu, Q. et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning, pp. PMLR, 2022.
Guan, C., Chen, F., Yuan, L., Wang, C., Yin, H., Zhang, Z., and Yu, Y. Efficient multi agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 35:1020-1033, 2022a.
Guan, C., Chen, E., Yuan, L.,, Wang, C., Yin, H., Zhang, Z., and Yu, Y. Efficient multi-agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 15:1000- 2022b.
Gupta, N. and Kahou, S. E. Cammarl: Conformal action modeling in multi agent reinforcement learning. arXiv preprint arXiv:2306. 1 11128, 2023.
He, H., Boyd-Graber, J., Kwok, K., and Daumé III, H. Opponent modeling in deep reinforcement learning. In International conference machine learning, pp. 1804- 1813. PMLR, 2016.
References
Alami, R., Fleury, S., Herrb, M., Ingrand, F., and Robert, F. Multi-robot cooperation in the martha project. IEEE Robotics & Automation Magazine, 5(1):36-47, 1998.
Hermandez- P., Kartal, B., and Taylor, M. E. Agent modeling auxiliary task for deep reinforcement learn- ing. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment, vol- ume 15, pp. 31-37, 2019.
Albrecht, V. and Stone, Reasoning about hypothetical agent behaviours and their parameters. In Proceedings of the 16th Conference on Autonomous Agents and MultiA- gent Systems, pp. 547-555, 2017.
Hu, H. and Foerster, J. N. Simplified action decoder for deep multi-agent reinforcement learning. In International Conference on Learning Representations, 2019.
Christianos, F., Schäfer, L., and Albrecht, Shared experi- ence actor-critic for multi-agent reinforcement learning- Advances in neural information processing systems, 33: 10707-10717, 2020.
Hu, H., Lerer, A., Cui, B., Pineda, L.,, Brown, N., and Foer- ster, J. Off-belief learning. In International Conference Machine Learning, pp.
PMLR, 2021.
Christianos, F., Papoudakis, G., Rahman, M. A., and Al- brecht, V. Scaling multi-agent reinforcement learning
Jeon, J., Kim, W., Jung, W., and Sung, Y. Maser: Multi- agent reinforcement learning with subgoals generated
10

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
from experience replay buffer. In International Confer- ence Machine Learning, pp. 10041-10052. PMLR, 2022.
Ma, M., Liu, Sokota, S., Kleiman-Weiner, M., and Foer- ster, J. N. Learning intuitive policies using action features. In International Conference on Machine Learning, pp. 23358-23372. PMLR, 2023.
Jiang. J. and Lu, Z. The emergence of individuality. In International Conference Machine Learning, pp. 4992- 5001. PMLR, 2021.
Ma. X., Yang, Y., Li, C., Lu, Y., Zhao, Q. and Yang, J. Modeling the interaction between agents in cooperative multi- reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 853-861, 2021.
Jo, Y., Lee, S., Yeom, J., and Han, S. Fox: Formation-aware exploration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- 2024.
Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019.
gence, volume 38, pp.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 2013.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap,
T., Harley T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928- 1937. PMLR, 2016.
Kontogiannis, A. and Vouros, G. A. Inherently interpretable deep reinforcement learning through online mimicking. In International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, pp. 160- 179. Springer, 2023.
Mordatch, I. and Abbeel, P. Emergence of grounded com- positional language in multi-agent populations. In Pro- ceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Li, C., Wang. T., Wu. C., Zhao, Q.. Yang, J., and Zhang, C. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991 2021.
Moreno, P., Hughes, E., McKee, K. R., Pires, B. A., and Weber, T. Neural recursive belief states in multi-agent reinforcement learning. arXiv preprint arXiv:2102.000 02274, 2021.
Li, J., Kuang, K., Wang, B., Li, X., Wu, F., Xiao, J., and Chen, L. Two heads are better than one: A simple explo- ration framework for efficient multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Nguyen, D., Venkatesh, S., Nguyen, P., and Tran, T. Theory of mind with guilt aversion facilitates cooperative rein- forcement learning. In Asian Conference on Machine Learning, pp. 33-48. PMLR, 2020.
Lin, X., Tang, Y., Lei, X., Xia, J., Zhou, Q. Wu, H., and Fan, L. Mare-based distributed cache placement for wireless networks. IEEE Access,
Nguyen, D., Le, H., Do, K., Venkatesh, S., and Tran, T. Social motivation for modelling other agents un- der partial observability in decentralised training. In Elkind, E. (ed.), Proceedings of the Thirty-Second In- lernational Join! Conference on Artificial Intelligence, IJCAI-23, pp. 4082-4090 International Joint Confer- ences Artificial Intelligence Organization, 8 2023, doi: 10.249633ijcca.22023454 URL https: / doii. orq/ 1 24.249666 NT..2963/ijjcm 2023 454. Main Track.
2019.
Liu, Z., Wan, L., Yang. X., Chen, Z., Chen, X., and Lan, X. Imagine, initialize, and explore: An effective explo- ration method in multi-agent reinforcement learning. In Proceedings of the AAA/ Conference Artificial Inteli- gence, volume 38, pp. 17487-17495, 2024.
Lowe, R., WU, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, Multi-agent actor-critic for mixed cooperative-competiivv environments. Advances in neu- ral information processing systems, 30, 2017a
Oliehoek, F. A., Amato, C., et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016
Papadopoulos, G., Kontogiannis, A., Papadopoulou, F., Pou- lianou, C., Koumentis, I., and Vouros, G. An extended benchmarking of multi-agent reinforcement learning algo- rithms in complex fully cooperative tasks. arXiv preprint arXiv:2502.04773, 2025.
Lowe, R., WU, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competiiie environments. Advances in neu- ral information processing systems, 30, 2017b.
Papoudakis, G. and Albrecht, S. V. Variational autoencoders for opponent modeling in multi-agent systems. In AAAI 2020 Workshop on Reinforcement Learning in Games, 2020.
Ma, L., Wang, Y., Zhong, F., Zhu, S.-C., and Wang, Y. Fast peer adaptation with context-awand aware exploration. In Forty- first International Conference on Machine Learning.

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Papoudakis, G., Christianos, F., Schäfer, L.,, and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint vrXiv:2006.07869, 2020.
In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085- 2087, 2018.
Tan, S. and Chen, B. Attentional opponent modelling for multi-agent cooperation. In 2023 International Joint Conference on Neural Nenworks (IJCNN), pp. 1-9. IEEE, 2023.
Papoudakis, G., Christianos, F., and Albrecht, S. Agent mod- elling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210-19222 2021.
Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Rahman, A., Bhattacharya, A.. Ramachandran, T., Mukher- jee, S., Sharma, H., Fujimoto, T., and Chatterjee, S. Ad- versar: Adversarial search and rescue via multi-agent reinforcement learning. In 2022 IEEE International Sym- posium on Technologies for Homeland Security (HST), pp. 1-7. IEEE, 2022.
Valiente, R.,
Toghi, B., Pedarsani, R., and Fallah, Y. P. Ro- bustness and adaptability of reinforcement learning-basee cooperative autonomous driving in mixed-autonomy traf- fic. IEEE Open Journal of Intelligent Transportation Systems, 3:397-410, 2022.
Raileanu, R., Denton, E., Szlam, A., and Fergus, R. Mod- cling others using oneself multi-as aagent reinforcement learning. In International conference on machine learn- ing, pp. 4257-4266. PMLR, 2018.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Rashid, T., Samvelyan, M., De Witt, S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234 7284, 2020.
Wen, M., Kuba, J., Lin, R., Zhang, W., Wen, Y., Wang. J., and Yang. Y. Multi-agent reinforcement learning is a sequence modeling problem. Advances Neural Infor- mation Processing Systems, 35:16509-16521, 2022.
Ruan, J., Du, Y., Xiong, X., Xing, D., Li, X., Meng, L.., Zhang, H., Wang. J., and Xu, B. Ges: Graph-based coor- dination strategy for multi-agent reinforcement learning. In Proceedings of the 21st International Conference Autonomous Agents and Multiagent Systems, pp. 1128- 1136, 2022.
Xiao, Y., Tan, W., and Amato, C. Asynchronous actor- critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 15:4385-4400 2022.
Shang, W., Espeholt, L., Raichuk, A., and Salimans, T. Agent-centric representations for multi-agent reinforce- ment learning. arXiv preprint arXiv:2104.094/ 2021.
Xu, P., Zhang, J., Yin, Q., Yu, C., Yang, Y., and Huang, K. Subspace-awar exploration for sparse-reward multi- agent tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11717-11725. 2023.
Son, K., Kim, D., Kang. W. J.. Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Inter- national conference on machine learning, pp. 5887-5896. PMLR, 2019.
Xu, Z., Bai, Y.. Li. D.,, Zhang, B., and Fan, G. Side: State inference for partially observable cooperative multi-agent reinforcement learning. In Proceedings of the 21 Inter- national Conference on Autonomous Agents and Multia- gent Systems, pp. 1 1400-1408 2022.
Su, J., Adams, S., and Beling, P. value-decomposition multi- agent actor-critics. In Proceedings of the AAAI conference on artificial intelligence, volume 35. pp. 11352-11360, 2021.
Yang. Y., Chen, G., Wang. W., Hao. X., Hao, J., and Heng, P.-A. Transformer- based working memory for multiagent reinforcement learning with action parsing. Advances in Neural Information Processing Systems, 35:34874- 34886, 2022.
Sun, J., Chen, S., Zhang, C., Ma. Y., and Zhang, J. Decision- making with speculative opponent models. IEEE Trans- actions on Neural Networks and Learning Systems, 2024.
Sunchag, P., Lever, G., Gruslys, A., Czameekk, W. M., Zam- baldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. "alue-decomposition networks for cooperative multi- multi-agent learning based on team reward.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooper- ative multi-agent games. Advances in Neural Informatin Processing Systems, 35:24611-24624, 2022.
12

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Yuan, L., Wang, J., Zhang, F., Wang, C., Zhang, Z., Yu, Y., and Zhang. C. Multi-agent incentive communication via decentralized teammate modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1466-9474, 2022.
Zhai, Y., Peng, P., Su, C.,, and Tian, Y. Dynamic belief for decentralized multi-agent cooperative learning. In Pro- ceedings of International Joint Conference on Artificial Intelligence, pp. 344-352, 2023.
Zhang, C. and Lesser, Coordinating multi-agent reinforce- ment learning with limited communication. In Proceed- ings of the 2013 international conference on Autonomous agents and multi-agen! systems, pp. 1101-1108, 2013.
Zhao, Q.. Zhu, Y., Liu, Z., Wang. Z., and Chen, C. Boosting value decomposition via umit-wise attentive state represen- tation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2305.07166 07,82, 2023a.
Zhao, Q.. Zhu, Y., Liu, Z., Wang. Z., and Chen, C. Boosting value decomposition via unit-wise attentive state represen- tation for cooperative multi-agent reinforcement learning. arXiv preprint 230500220.. 2023b.
Zhao, X., Pan, Y., Xiao, C.. Chandar, S., and Rajendran, Conditionally optimistic exploration for cooperative deep multi-agent reinforcement learning. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI '23. JMLR.org, 2023c.
Zheng, L., Chen, J., Wang, J., He, J., Hu, Y., Chen, Y., Fan, C., Gao, Y., and Zhang, C. Episodic multi-agent reinforcement learning with curiosity- exploration. Advances in Neural Information Processing Systems, 34: 3757-3769, 2021.
Zhu, C., Dastani, M. and Wang, S. A survey of multi- agent reinforcement learning with communication. arXiv preprint wwXii:2203.08975, 2022.
Zintgraf, L.,, Devlin, S., Ciosek, K., Whiteson, S., and Hof- mann, K. Deep interactive bayesian reinforcement leam- ing via meta-learning arXiv preprint arXiv:2101.0386. 2021.
13

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
A. Further Related Work
MARL and Exploration. In sparse rewards settings, MARL exploration strategies encompass density -based (Tang et al., 2017; Zhao et al., 2023c; Jo al., 2024), curiosity-driven (Zheng et al., 2021; Li al., 2024), and iformation-theoree (Li al., 2021; Jiang & Lu, 2021) approaches. MAVEN (Mahajan et al., 2019) aims agents to explore temporally extended coordinated strategies. However, it does not encourage exploration of novel states and the inference of the latent variable still needs to explore the space of agents" joint behaviors. (Nguyen et al., 2023), which only considers single learnable controller setting, employs social intrinsic motivation based on AM but relies on a priori knowledge of observation feature representations to compute intrinsic rewards. In contrast, SMPE uses the informative state embeddings to encourage an adversarial type of multi-agent exploration through a simple intrinsic reward method.
B. Results on MPE's Double Speaker Listerer
Due to page limit, present the results on Double Speaker Listener in this section of the Appendix. As be clearly shown, SMPE outperforms the other evaluated methods resulting in better joint policies. Interestingly, SMPE consistently demonstrates average episodic reward better than the other methods and converges to optimal policies.
Double Speaker Liseenrr
-00
300
Episodde Deward
-40
-50-
-60
SMPE
MAAZC
NAPPO
-70
ATM
-80
0.0
0.2
0.4
0.6
0.8
1.0
Steps
107
Figure 8. Results on Double Speaker Listener
14

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
C. Extended Preliminaries
C.1. Aueti-Agent Actor Critic (MAA2C): Independent Actors with Centralized Critic
We consider the CTDE version of the MAA2C algorithm with independent actors and centralized critic, which has been widely used in MARL literature (Papoudakis et al., 2020; Xiao et al., 2022; Christianos et al., 2020: Su al., 2021). The basic idea is that MAA2C constitues simple extension of the single agent A2C algorithm (Mnih al., 2016) the CTDE schema. The actor network of agent parameterized by or, is conditioned only the local observation history h4. The critic network, whose parameters we denote by &, uses the full state, 8;, of the environment and approximates the joint value function V The actor network is optimized by minimizing the actor-critic policy gradient (PG) loss:
Lewcor - log Ta, (a) hb) (ri+
- VV VV (Su4iiE)) 3 H (To, (a) h2))
(8)
where H the traditional entropy term of policy gradient methods. The critic network is trained by minimizing the temporal difference (TD) loss, calculated follows:
Lenti (k) = (V* ks;
-yy)
(9)
where y¹ r + (8)41;k) the TD target. We note that both the actor and the critic networks are trained on-policy trajectories.
C.2. MIIAM: Multi-Agent Local Information Agent Modelling (LIAM)
In the original paper (Papoudakis et al., 2021), LIAM is an agent modelling method controlling one agent and assuming that the rest -i modelled agents take actions based on some fixed set of policies 11. MLIAM is our custom implementation for the multi-agent extension of the single-agent method, LIAM: MILAM assumes that all agents are learnable and homogeneous (i.e. they share their policy parameters). MLIAM's algorithm is similar to LIAM's, ss with the only exception that the following now hold for every agent, instead of only the single controlled agent. It assumes the existence of a latent space 2 which contains information regarding the policies of the other agents and the dynamics of the environment. Aiming to relate the trajectories of each agent i to the trajectories of the other agents -i utilizes an encoder-decoder framework. Specifically, timestep the recurrent encoder fü 7 + 2 takes as input the observation and action trajectory of the agent i until that timestep (of.: 1) and produces an embedding z/. Then the decoder fü - --i takes the embedding 21 and reconstructs the other agents' observation and action (o, aa, The encoder-decoder loss of agent for a horizon H is defined as:
H LED El(t - 11 log fi. (a, =t)]
(10)
where 2/ = fú a/6-1) and fir fi. denote the decoders' output heads for the observations and actions respectively. The learned embedding is used additional input in both the actor and the critic considering augmented space Orong O 2, where the Oi is agents' original observation space and 2 is the embedding space. Considering an A2C reinforcement learning algorithm and a batch of trajectories B the A2C loss of agent i is defined as:
Tiaz0 Bi Vo((5111 Vo(s))) Ålog mo(af oe zt) Bu Bn (To(aa of,4
(11)
where H is the entropy and /3 a fixed hyperparameter contrrllin the initial random policy.
D. Missing Proof
Proof of Proposition 2.1
15

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Proof. Let The be the joint policy parameterized by the joint parameters V. Then, we have:
20
SM max max E Eepuu Barre PPr (=))
Σ
rrr
max B
Σ
,'1t
2
Σ
max B
0
V
where the inequalities hold because θ C. V and thus every Th solving the Dec-POMDP problem, could be equivalent with some Ty (e.g., when Ty completely ignores the latent variable Pw). On the other hand, V is the optimal state value function and thus holds that VšM= S V* which implies that VSM V*.
E. Extended Experimental Setup
The selected tasks from all three evaluated benchmark environments, namely MPE, LBF and RWARE, are partially observable and rely on the history of observations the learning agents.
E.1. Multi-agent Particle Environment (MPE)
We used the source code in http://gghhbbbooooommmmmllmmlllh (under the MIT licence).
E. I.I. SPREAD: A COOPERATIVE NAVIGATION TASK
Figure Spread: A cooperative navigation task
This fully cooperative environment involves N agents and landmarks, with the primary goal of enabling agents to learn how to cover all landmarks while avoiding collisions. key challenge is balancing global and local incentives: agents rewarded based the proximity of the closest agent each landmark incurred for collisions. This structure creates trade-off, requiring agents to coordinate effectively to minimize distances to landmarks while also avoiding collisions in shared space. The dual challenge of learning collision avoidance and optimal landmark coverage in a multi-agent context demands the development of cooperative strategies, making this task particularly complex for multi-agent reinforcement learning (MARL), especially as N increases. Insights derived from this environment have practical applications in traffic management systems, distributed sensor networks, urban planning, and smart citics. The environment is depicted in Figure
global scale, while local penalties
Observation space: Agents receive their current velocity, position, and the distance between landmarks and other
16

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration agent positions their input.
Observation size: [24.]
Action space: The action space is discrete and involves 5 actions: standing still, moving right, left, up and down.
Reward: All agents receive the same team reward, which includes the summed negative minimum distance to any other agent. Additionally, the collisions between any two agents are with a reward of -1.
In our experiments, the scenario spread-n means Spread with n agents and n landmarks.
E. 1.2. DOUBLE SPEAKER -ISTEEER COOPERATIVE COMMUNICATION TASK
Figure 10. Double Speaker-Listener: cooperative communication task
This fully cooperative environment features a Speaker and a Listener, where the Speaker is responsible for effectively communicating information about target landmark while the Listener navigates based on the guidance provided by the Speaker's signals. Operating in partially observable setting, this environment introduces challenges such communication complexity and the need for extensive coordination despite limited perspectives and feedback. The insights derived from this environment have valuable applications in robotic teams, human-robot collaboration, and autonomous navigation systems. The environment is depicted in Figure 10.
The speaker agent observes only the colour of the goal landmark which is represented three numeric values. The listener agent receives observations its velocity, relative landmark positions as well as the communication of the speaking agent.
Observation Space:
Observation size: [11.]
Similar to other MPE environments the listener's action is space is discrete and includes five actions: (standing still, moving right, left, up and down). The speaker's action space is also discrete however it includes three actions to communicate the goal the listener.
Action Space:
Reward: The reward is common for both agents and it is calculated as the negative square Euclidean distance between the listener's position and the goal landmark's 's position.
E.2. Level-based Foraging (LBF)
We used the source code in
(under the MIT licence).
The Level-Based Foraging (LBF) (Albrecht & Stone, 2017) benchmark offers fully cooperative environments where agents navigate a two-dimensional grid-world to collect food items. Each agent and food item are assigned specific level, and agents can move in four directions on the grid, collecting food only when the combined levels the agents meet or exceed that the food item. A significant challenge within the LBF environments is the sparse reward structure, as agents must coordinate effectively to simultaneously consume specific food items. The insights derived from this environment have
17

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Figure 11. Lovel-based Foraging
practical applications in multi-robot collaboration, resource management in supply chains, and coordination during disaster response efforts. The environment depicted in Section 4.2.
Observation space: All agents receive triplets of the form (z, level) observations. The number of triples is equal to the sum of the number of foods and the number of agents in the environment. The observations begin with the food triples and are followed by the agents triples. Each triplet contains the x and coordinates and level each food item or agent.
Observation size: 3 x (number of agents number of foods)
Action space: The action space is discrete and involves six actions: standing still, move North, move South, move West, move East, pickup.
Reward: In Level-Based Foraging environment agents are rewarded only when they pick up food. This reward depends on both the level of the collected food and the level of each contributing agent, and it is defined for the agent as follows:
FoodLevel AgentLevel
Σ FoodL.evels Σ LoadingAgentsl.evel
Also rewards are normalized in order the sum of all agent's returns on solved episode to equal one.
Last but not least, the LBF task under the name PS-Gx--Pp-FF-coopp" corresponds to the cooperative task that has: an GxG grid consisting of P agents, F foods and with partial observability within a window of length for each agent.
Remark E. 23-11x11-3p-2f and 4s-11x11-3p- are the most difficult task among the selected LBF ones, because: (a) a high-value state associated with some positive reward practically impossible to be reached through an initial random policy (because a unique combination of agents must participate in the collection specific food), and (b) selecting actions that lead such states very difficult due the excessive partial observability of this task. Our results totally agree with the above facts, no other method manages to find valuable states the 11xll tasks, thus resulting only in zero episodic reward over time.
E.3. Multi-Robol Warehouse (RWARE)
We used the source code in http:///gthabbbcooommoeeeeeeeste
(under the MIT licence).
The Multi- Robot Warehouse (RWARE) (Papoudakis al,, 2020) environment models fully cooperative, partially observable grid-world in which robots are tasked with locating, delivering, and returning requested shelves workstations. Agents navigate within grid that limits their visibility, providing only partial information about nearby agents and shelves, which makes their decision celssion-mate making processes more complex. One of the primary challenges in this environment the sparse reward structure; agents receive rewards only upon successfully delivering shelves, requiring them to follow a specific sequence of actions without immediate feedback. This situation demands effective exploration and coordination among
18

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Figure 12. Multi-Robot Warehouse (RWARE)
agents to achieve their objectives. The insights gained from RWARE have meaningful applications logistics and warehouse management, where multiple autonomous robots must work together to optimize inventory handling and order fulfillment. The tasks in RWARE vary in the world size, the number of agents, and shelf requests. The default size settings in the Multi-Robot Warehouse environment have four options: "tiny," "small," "medium," and "large." These size specify the number of rows and columns for groups of shelves in the warehouse. In the default setup, each group of shelves comprises 16 shelves arranged in an 8x2 grid. The difficulty parameter specifies the total number of requests at time which by default equals the number of agents (N). One can modify the difficulty level by setting the number of requests to half ("-hard") or double ("-easy") the number of agents. The sparsity of rewards and high-dimensional sparse observations make RWARE challenging enviromment for agents, they need to perform series of actions correctly before receiving any rewards. The environment depicted in Figure 12.
The environment is partially observable, and agents can only observe 3x3 grid containing information about surrounding agents and shelves. Specifically the observations include the agent's position, rotation, and whether is carrying a shelf, the location and rotation of other robots, the shelves and whether they are currently in the request queue.
Observation space:
Observation size: [71.]
Action space: The action space is discrete and involves five actions: turn left, turn right, forward, load/unload shelf.
Reward: The agents receive rewards only when they successfully complete the delivery of shelves.
19

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
E.4. Extended Ablation Study
E.4.1. COMPARISON OF SMPE TO SMPE(no
1oh-22 1at-a-9999-p-22--c0pp
0.3
0.
0.5
tpbrck
1.5
0.5
1.0
1.0
Stato
Steps
Figure 13. Comparison of SMPE to SMPE(no_jnrr) on the three most difficult evaluated LBF tasks
E.4.2. FURTHER RESULTS THE COMPARISON SMPE TO MLIAM
In this subsection of the Appendix, provide more results the comparison of our SMPE? with MLIAM in Figure 14.
Double Speaker-Listener
Spread-44
Iof BwD--pp----pp-- 20--hhooopp
-550
-000
-022
tslaak hoourd
0.8-
-420
-440
0.4
-050
-400
-500
1.0
267
Iof-29- 2x1 2-222222p--22 2-12--2----oooppp
Iof-7s-20x20- Sp-3-coop
1bf-24-114113ppp22222 3p3-2---pppp
Detsar Benurt
0.6-
01.-
0.1
0.4
0.2
0.2
SMPS
Figure 14. Further ablation study results the comparison between SMPE and MLIAM on MPE and LBF
20

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
E.4.3. WHY DO WE NEED SECOND CRITIC FOR TRAINING w' W.R.T. POLICY OPTIMIZATION?
In this subsection of the Appendix, study why SMPE really needs critics: one (E) for providing the advantages needed for training the actor and one (k) needed for training w' with respect to policy optimization. As be clearly seen from Figure 15, the selection of two critic models is essential for the training of SMPE? the bascline (no_standardecritt) which uses only the critic k for both purposes displays both worse average episodic reward and convergence. More specifically, it suffers from high variance, attributed to the fact that in some runs it entirely diverges from good policies leading to very poor performance.
8x8
3.0
-3500
-200
-6400
044
0.4
-0900
622
-0502
3.0
0.6
167
166
161
Figure 15. Ablation study why need the second critic model: Baseline SMPE (no standard critic) is our method using the critic k that the one conditioned on w also for providing the advantages needed to train the actor.
E.4.4. ABLATION STUDY ON AREC
Figure 16 illustrates the importance of Ave to the contribution of the reconstruction loss to the overall objective in a Spread task. As can be clearly seen from the plot, lowering the value of Aree worsens both convergence and average episodic reward in this task. We attribute this to the fact that lowering the value of Are leads to worse state modelling, and thus the agents are not capable inferring good beliefs about the unobserved state. Interestingly, as we increase the value of Aree the overall performance deteriorates slightly, while displaying higher variance. This because the reconstruction term is penalized more in the overall objective, dominating the policy optimization term.
spread-4
9900
Epcooii Howard
11000
SMPE? ([rec__3 2)
-1700
SWPP rec 0.11
SMPE? [[rec_o.00)
-1300
SMPE? (Lroc 30)
SMPE? (Lrec_100) 2001
0.0
0.2
0.4
0.6
0.8
1.0
Staps
167
Figure 16. Ablation study Ave
21

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
E.4.5. ABLATION STUDY ON NNORM
Regarding Anorm, the ablation study showed that if equals O, then it can negatively affect the algorithm performance. because, feasible solution for w/ is to converge to O. Here, we provide further ablation study in Figure 17, which shows that if Anomm is set to value ranging from [0.1, 1], then performance is pretty much the same.
Spread-4
lof. 11.111.11 11111-3p--2-aaae 22-p2p5
-200
-1000
Epioedi Roward
0.4
1200
-1400
0.0
0.4
0.6
0.0
0.6
Staps
Stegs
167
Figure 17. Ablation study on Anumn
E.4.6. RESULTS ON LEARNING THE STATE BELIEF REPRESENTATION WITH RESPECT POLICY OPTIMIZATION
In this subsection of the Appendix, provide more results the significance to learn the state belief representations with respect to policy optimization in Figure 18. As we discussed in Section 4.4, learning the state belief representations with respect to policy optimization can yield significant improvements in terms of average episodic reward and convergence.
wware-ing-Dat--een
reote-liny-faggheee
10-
Eptoodi ic Reward
Eptoodt Rewword
-000
Estoodic Reward
-903
0.4
-1000
-1190
0.8
Steps
(no.
Stapss
Figure 18. Ablation study on learning the state belief representations with respect to policy optimization
E.4.7. STUDYING THE DYNAMIC NATURE OF THE INTRINSIC REWARD
In this subsection of the Appendix, we examine the dynamic nature of the intrinsic rewards proposed in SMPE As highlighted in Section 3.2, to enhance stability, we make z more stable by avoiding significant changes the parameters (wi, (b)) between successive training episodes. To achieve this, we perform periodic hard updates of wi and Φ; using fixed number of training time steps as the update period. Figure 19 shows that the intrinsic rewards do not fluctuate but rather decrease smoothly throughout training until they reach minimal plateau. Such behavior is what expected: the agents to explore the state space and gradually to converge to more deterministic optimal policies.
22

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Intrinsic Rewards time lbft 2: 2p-13-113p-22
0,77
0,05
0,05
0.04
0.03
0.02
692
0.8
1.0
Stops
Figure 19. Ablation study: Studying the dynamic nature of the intrinsic reward
Embeddings
Accuracy
57.5% %
2¹ (without KL)
99.3%
2¹
(without w²)
80.8%
Table Quantitative ablation study on the embeddings z' in an IBFF task at 45th time step: Logistic Regression on 5-fold cross validation using the t-SNE representations the training data and the agent IDs as the labels.
E.4.8. FURTHER T-SNE VISUALIZATIONS AND ANALYSIS THE STATE BELIEF EMBEDDINGS
VAE VREVeukrro d'btaidrggg Warkthood Sherse
LOST Wessbarten shording bolt
Figure 20. SMPE
Figure 21. SMPE? (LkL = 0)
We trained logistic regression classifier using t-SNE representations as training data and agent IDs as labels, and performed standard 5-fold cross validation. Table validates that SMPE?'s beliefs are cohesive, as they are well entangled and difficult to separate (57.5% accuracy). In contrast, SMPE? without either T.L or forms much less cohesive beliefs, as it achieves perfect accuracy (99.3%) or high accuracy (80.3%), respectively. We note that the perfect score of SMPE without LKI. due to the fact that if the agents' beliefs do not share the same prior, then they can be formed in totally different (and easily separated) regions. The above results also validate the importance w" for producing more cohesive z!.
E.4.9. IS SMPE2 FLEXIBLE? CAN WE USE MAPPO AS THE BACKBONE ALGORITHM?
To address question Q6, we evaluate SMPE using MAPPO as the backbone algorithm. We denote this method by SMPE_PPO. Figure 22 highlights the flexibility of our method, as SMPE_PPO significantly outperforms its backbone counterpart all benchmarks. Remarkably, SMPE PPO almost perfectly solves the 9x9-3p-2f task, with the backbone MAPPO demonstrating very poor performance.
23

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
spread-3
10-22 (122--x8-5p-2-co
BF-25-000 3o-20000-322-c00
rwore-dny-2og-hace
402
2.6
0.3
302
-552
0.2
244
0.8
6.
100
2.0
0.2
0.1
0.6
0.3
0.2
0.2
044
S0ao
Staps
Siae
Figure 22. SMPE2 flexible: SMPE_PPO against its backbone algorithm MAPPO
E.4.10 10. FURTHER ABLATION STUDY ON THE EXPRESSIVENESS oF z LOSS CURVES
2f-coop
2000
250
Sen SenSppereee Recendrictnn Lo5
1730
200
1500
1250
1000
100
750
500
250
0.2
0.6
1.0
0.0
0.4
1.0
167
167
Figure 23. Self-supervised reconstruction loss on LBF
Figure 24. Self-supervised reconstruction loss on Spread
Figures 23 and 24 show that the reconstruction loss is decreasing throughout the training of SMPE2 and thus = conditioned on w is able to reconstruct informative features (identified by w²) of the global state.
24

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
E.4.11. EXTRA ABLATION STUDY ON THE EXPRESSIVENESS AM FILTERS wo
AM Fitters correspondigg to
LBF: 4p-11xt1-33p-2f
Food (right)): Postiionx (8.91) Posilionyy (0.90), Level (0.ro)
Food (left): Position xx (0.08) Posttion (0.06), Level (0.57)
Our interpppeetti:::::
Agent Positionxx (0.27). Positionly (0 (9.36), Lo.36) (0.47)
Informative fnaturas that faciliiate reconstruction:
Agent 2: Fosttion x (0.30), Position y (0 (0.92) Level (0.49)
(2)
Postioonx 97), Position
Level (0
Informative features that facilitaee pollcy optimization:
AM Filters correponning to Agent
(Left)
(Right)
Food (right:: Poaltionu (6 Poakionny (0. 94) Level (0.70)
The remaining features are filtered by the titers (considering threshold of 0.40)
Food (eft): Positionxx x (0.10) Positionyy (0.12), Level 0.58)
Agent Fostiion
Posttion (0.91), Level
Agent 2. Fosttion x (0 29), Posttion.y (0.34). Level (3.47)
Agent Position.x Posttion /0 Level
Ageit i within the eight of Agent 1,
Player Level Player 2: Level 1
Agent position and
Player 3: Level
AM Filters corresponding Apent 2
AM Fillers corresponctng Agent 3
Food (right): Position.x (0.1 12), Posiiion.y (0. 14), (0.4e
Food (right): Postioo Fosition (0.58) (0.66)
Food (left): Position_a (0.01), Position,y (0.01), Level (o. (3)
Food (left): Positionxx (0.07), Position_y (0.05), Level (0.01) 01)
Agent 1: Fositionxx (0. 94), Posttion.y (0.3 55),5, Level (0
Agent Position,x (0.41), Position.y (0.32),
64)
Agent Position,x x (0.05). Posttion.y y (0.72) Level (0.
Agent 2:
Position_x (0. (0)70), Poattion_y (0 98), Level (0:57)
Agent Fosition (0.02), Posttion.y y (0.01), Level 07)
Agent 3: Postonn (Q.9ti)
Position y (0.78), Level (0.79)
Figure 25. Ablation study: Goal: The optimal policy (found by SMPE*) is all players to eat the right food. Agent1: Does not know where both the right food and agent3 are located, or their levels, and seeks to find them. Moreover, it seeks to find Agent2 sees the right food, Agent2: Has filters very similar Agentl (as expected). Agent3: Knows where the right food is located and its level, and seeks to find if the others know it as well and have the required levels for it, as well as the others are near to the left food. It has very similar filters for both Agentl and Agent2 (as expected).
In the cooperative LBF task of Figure 25, all players must eat at the same time food, whose level equals the sum of the players' levels. Each player has a limited sight range of boxes.
Figure 25 demonstrates a global state of the LBF task, along with the AM filters w' of agents (bottom) and 3 (top). In this state, agentl and agent2 observe each other (i.e., their locations and levels), while agent3 observes food (right) (i.e., its location and level). The figure is instance of the best SMPE? policy, which guides all players to eat food (right) the same time. In the figure, the features that are not highlighted (with blue/orange) are filtered out as uninformative.
The figure illustrates the expressivity of w'. It shows that uninformative state features are blocked by w by assigning them low weights, if either: (a) are easily inferred through reconstruction but are redundant for enriching 2² (e.g., Agent2/Position. x (0.05) see the bottom left filters), or (b) do not help policy learning (e.g., Fooddriihtt/Position.. (0.12) see the bottom left filters). More specifically:
Regarding Agentl, the agent does not know where Agent3 and food (right) are located, or their levels, and seeks to find them (see the values highlighted with blue). In doing so, the AM filters of Agentl corresponding to both Agent2 and Agent3 block information about food (left) since it irrelevant to its policy. Also, the filters corresponding to Agent2 block information regarding both Agent3 and the location of food (right) because they are not observed by Agent2. Moreover, these filters block Agent2/Position.x but not Agent2/Position..y because Agent1 and Agent2 have the same x-coordinate. We dismissed the AM filters of Agent2, as they were pretty much the same those of Agent1.
Regarding Agent3, the agent seeks to find if the others know as well where food (right) is located, and if they have the required levels for it, as well as if the others observe food (left) (see the values highlighted with blue). Its AM filters block (a) the features that describe the location of food (left), and (b) the positions of the other agents. As for the latter, Agent3 is already near food (right), which is the target of the joint policy, and waits for the other agents to appear on its sight in order to eat food (right) at the same time. Moreover, the filters corresponding to both Agentl and Agent2 are very similar, as expected, because both agents have the same levels and are located next to each other.
25

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
We highlight with orange the high-value features which intuitively were selected by SMPE facilitate the reconstruction training, helping 2² incorporate useful information about the neighhorhood of the agent.
E.4.12. COMPARISON OF RUNNING TIMES OF THE EXAMINED METHODS
In Table 2, we compare the running times of the examined methods in LBF: 1s-11x11-2p-22 using the following specs: GPU RTX 3080ti, a CPU 11th Gen Intel(R) Core(TMM) 19-1 1900 and a 64 GB RAM.
Remark E.2. In experiments, SMPE is faster approximately 25 times than MASER, 30 times than EMC, 17 times than EOI and 2 times than ATM in
MARL Method
Running Time
MAA2C
Od 00h 37m
COMA
Od 00h 54m
ATM
Od 02h 32m
EOI
Od 17h 11m
EMC
1d 05h 34m
MASER
1d Olh 08m
SMPE
Od 01h 13m
Table 2. Comparison of running times of the examined methods on 1bf:2s-12xx1-22
26

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
F. Implementation Details
Our implementation built on top the EPyMARL' (under the MIT License) python library. For all baselines, we used the open source code of the original paper. SMPE² uses the standard MAA2C urchitecture (see EPyMARL) and three-layer MLPs for both the Encoder-Decoder (ED) and for the AM filters. It is worth noting that for ED in our experiments we did not observe any further improvement in performance when using standard RNNs (such GRUs, LSTM) instead of MLPs. In practice, we update ED parameters every NED steps. Last but not least, for all benchmark settings, agents (for all algorithms) used shared policy parameters, except for Spread 3-4-5 where each agent had its own parameter weights.
Name
Description
Value
H (mpe-lbf-rware)
time horizon
10M-10M-40M
Nenvs
number of parallel envs of MAA2C
10
Nen
(mpe-Ibf-rware)
maximum length of an episode
25-50-500
Nup
number of environment steps before optimization
100
ND
replay buffer capacity (for ED training)
50,000
Nbs
ED replay buffer batch size
16
NED
update ED parameters every NED time steps
2000
Nutup
update target filter parameters every Nutup time steps learning rate for RL algorithm learning rate for filter update learning rate for filter update in critic loss learning rate for ED hidden dimensionality of Actor and Critic NN
100,000
lr
0.0005
Im (rware)
0.0005 (0.00005)
Ir Iuriiiic (mpe, rware) IgD
0.00005 (0.0000005)
0.0005
hidden.dim dim (rware)
128 (64)
latent dim (mpe)
latent dimensionality ED
32 (64)
?
discount factor
0.99
Aree (lbf)
lambda coefficient for Lue lambda coefficient for LKL lambda coefficient for T
(0.5)
ÀKL
0.1
Amoem (mpe-Ibf-rware)
0.1-1-0.1
3H
entropy coefficient in policy gradient intrinsic reward coefficient shared policy params shared ED params
0.01
B (Ibf-rware)
0.1-0.001
shared (mpe-spr8-Ibf-rwee) -bf-rwarrr
F-T-T-
shared_E]
F (False)
Table 3. Implementation Details of SMPE
OOur official source code can be found at http://gittuh..om/dddeeaa/a/s// see http://guhhb..cmmoeeeggn///gggg
27

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Name
Value
batch size
10
hidden dimension
64
learning rate
0.0005
reward standardisation
True
network type entropy coefficient
GRU
0.01
target update
200
buffer size
10
0.99
observation agent id observation last action
True
True
n-step epochs
5
4
clip
0.2
Table Hyperparameters for MAPPO
Name
Value
batch size hidden dimension learning rate reward standardisation
32
64
0.0005
True
network type evaluation epsilon
GRU
0.0
epsilon anneal epsilon start
50000
1.0
epsilon finish
0.05
target update buffer size
200
5000
0.99
observation agent observation last action episodic memory capacity episodic latent dimension soft update weight weighting term À of episodic loss
True
True
1000000
0.005
0.1
curiosity decay rate (n)) number of attention heads attention regulation coefficient mixing network hidden dimension
0.9
0.001
32
hypernetwork dimension hypernetwork number of layers
64
Table Hyperparameters for EMC
28

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Name
Value
optimizer
RMSProp
hidden dimension
64
learning rate reward standardisation
0.0005
False
network type evaluation epsilon
GRU
0.0
epsilon anneal
50000
epsilon start
1.0
epsilon finish
0.05
target update
200
buffer size
5000
0.99
observation agent id observation last action
True
True
mixing network hidden dimension representation network dimension
32
128
Q
0.5
A
0.03
À
0.0008
AE
0,00006
AD
0.00014
Table Hyperparameters for MASER
Name
Value
optimizer
Adam
batch size
10
hidden dimension
128
learning rate
0.0005
reward standardisation
True
network type
GRU
target update buffer size
200
10
0.99
observation agent id observation last action
True
True
n-step entropy coefficient
5
0.01
classifier (Φ) learning
0.0001
classifier (Ø) batch size
256
classifier (82)
0.1
Table 7. Hyperparameters for EOI
29

---

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
Algorithm State Modelling for Policy Enhancement and Exploration (SMPE*)
1: Initialize Nenus parallel environments (required for the backbone MAA2C)
2: Initialize actor networks with random parameters 1/1...., V
3: Initialize the critic networks and their target networks with random parameters k, & and k', 5
4: Initialize N encoder-decoder networks with random parameters dI..... UN and , WN
5: Initialize a replay buffer D of maximum capacity ND
6: Initialize N weight networks with random parameters 6 ..... ON
7: Initialize N target weight networks (for w²) with random parameters (ØT).... ..... (
8: for time step t = ,,,,,..4 H do
9: for agent = 1,...,... ,....,N N do
10: Receive current observation of
Sample belief z ~ 2o. (z* of) Sample action aj from Te (a) hgg =f)
13:
end for
14:
Execute actions and receive states 8t+1, observations 01+1 and the shared reward rt
15:
for agent = 1, ,,...,N N do
16:
Calculate the intrinsic reward ft and compute total reward 7/ = r2 + 38f using the SH function, and use of as the reward of agent
end for Store the received (joint) transition (st, a, or. P1,1,, 0141, in if Parallel Episodes Terminate then
20:
for agent NN do
21:
Update v/; on the sampled trajectories by minimizing Lactor of the Equation (8)
22:
Update on the sampled trajectories by minimizing Lenue of the Equation (6)
23:
Update o¹" and & on the sampled trajectories by minimizing T of the Equation (6)
24:
end for
Restart a new episode for each of Nenvs parallel environments
26: end if
27:
if condition for training the encoder-decoders. is met (NED) then
28:
for agent ,,...,N do
29:
Update wi on D using batch size No. by minimizing L,ee (Equation (3)) and LKL (Equation (5))
30:
Update Φ; on 1 using batch size No. by minimizing Tree (Equation (3))
31:
Update o.eh on D using batch size No. by minimizing Liee (Equation (3)) and Lnorn (Equation (4))
end for
end if
34:
if condition for updating the target policy networks is met (Ntup) then Update ç' E and k' k
35:
36:
end if
37:
if condition for updating the target filter networks is met (N (Nwtup) then
38:
for agent = .....,,,, N do
Update (6) <5 end for
41: end if
42:
end for
30


Reviews:
[{"review_id": "qKoCc3hyQz", "reviewer": "Reviewer_W3Cu", "summary": "This paper proposes State Modelling for Policy Enhancement through Exploration, a novel approach to cooperative multi-agent reinforcement learning in partially observable environments without communication. The method enables agents to infer meaningful belief representations about unobservable states through variational inference and self-supervised learning, while filtering out redundant information. The authors claim to enhance agents' policies both explicitly by incorporating these beliefs into policy networks and implicitly through adversarial exploration. Experiments across three benchmarks demonstrate that the proposed consistently outperforms state-of-the-art MARL algorithms, particularly in cooperative tasks that require extensive coordination and exploration.\n\n## update after rebuttal\n\nI have interacted with the authors and my concerns were addressed. I think it is really important to add more details addressing the strength of contribution concerns by the other reviewers for the camera ready version in case of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and input, along with the positive evaluation. We respond to your comments and questions below.\n\n> 1. A more detailed analysis in the style of rliable [42] could be more informative than some subfigures in Fig 3 and 4. \n\nThanks for this comment. We will consider using rliable in the camera-ready version.\n\n> 2. Scalability with more agents\n\nRegarding the scalability of our algorithm, Figure 2 shows that it clearly outperforms the other baselines in MPE Spread with 8 agents, albeit by a smaller margin compared to MPE Spread with 3, 4, and 5 agents. Furthermore, additional evaluations in LBF  8s-25x25-8p-5f and 7s-30x30-7p-4f (tasks have been benchmarked by the AAMAS2025 paper [1]) demonstrate that our algorithm achieves superior performance compared to the others.\n\n|Env |SMPE |MAA2C |MAPPO|MASER|EOI|ATM\n|-----------------:|---------:|--------:|----------:|----------:|------:|-----:|\n| 8s-25x25-8p-5f| **0.64 +- 0.12**|0.52 +- 0.24|0.41 +- 0.23|0.01 +- 0| 0.07 +- 0.02 | 0.36 +- 0.28 |\n| 7s-30x30-7p-4f| **0.74 +- 0.02**|0.71 +- 0.02   |0.57 +- 0.03|0.01 +- 0| 0.04 +- 0.01 | 0.59 +- 0.04 |\n\n> 3. Comment on missing related works\n\nThank you for this comment. Does the reviewer some specific reference in mind? We will be happy to add missing related works. \n\n> 4. Figure 1 is not entirely clear and the arrow convention is not sufficient to distinguish the gradient and the data flow. \n\nThank you for this comment. Our overview of SMPE^2 and the arrow convention follows a visualization approach similar to that of many well-established MARL papers (see [43], [44], [46]). If the reviewer still has objections about the figure, we are open to discuss possible improvements. \n\n> 5. Justification on how the baselines are selected, especially the ones that are fairly simpler than the proposed method.\n\nOur set of baselines includes some of the most well-known and relevant algorithms to ours. The choice of simpler algorithms was intentional, as MAA2C serves as the backbone of our algorithm, making it a natural baseline. Additionally, while MAPPO is a generalization of the single-agent PPO algorithm, it achieves strong results in our benchmarks, as described in [47].\n\n> 6. The choice of t_SNE seems somewhat arbitrary?\n\nWe kindly disagree with this comment. t-SNE is a well established method used widely in the MARL community for embedding visualization (see [43],[44],[45]).\n\n> 7. Are all the timesteps pooled together for the classification or is the accuracy reported for each timestep separately?\n\nThanks for this comment. Indeed all the timesteps pooled together for the classification. \n\n> 8. The timestep 45 makes sense but it can also come across as cherry-picked\n\nThanks for this comment. The timestep 45 was not cherry-picked but selected on purpose to be at the end of the trajectory to represent the belief of each agent closer to the solution of the task. Furthermore in the Appendix E.4.8 we provide similar embedding visualizations for the timestep 30. \n\n> 9. Note that the authors mention the same work twice for no reason as two separate entries. Line 262 first column has a typo: “By doing do”. The Guan et al. paper is mentioned twice in the references. Line 1194 should be well-entagled instead\n\nThanks for your comments. We will fix these minor typos.\n\n**Due to space constraints, below we include responses to some of the questions of the other reviewers.**\n\n> 10. Difference from LIAM, MAVEN and [3]\n\nWe respectfully disagree with the reviewer's comment. \n\n- Regarding LIAM, please see lines 669-674. Also, note that LIAM does not use AM for exploration which is a major contribution of our paper.\n\n- Regarding MAVEN, some clear differences are the following: \n--\tMAVEN does not account for non-informative state information which can be detrimental to MARL performance.\n--\tOur method is not based on committed exploration but on an adversarial competition among the agents, guided by z_i and a count-based method.\n--\tOur latent variables are not shared but are unique to each agent.\n--\tOur z_i is used as input to each agent's policy, rather than a component of the joint action-value function.\n\n- Regarding [3], first of all, we believe that it is irrelevant as it solely considers single-agent RL, and thus it does not have to deal with MARL challenges, including multi-agent decentralized execution and non-stationarity. Additionally, [3] employs an intrinsic reward based on the prediction error of embeddings of pixel-based states, aiming to identify novel pixels. In stark contrast, we use a count-based hashing method on the belief z_i, which is informative of the unobserved state and policy optimization, ultimately improving joint exploration.\n\n[3] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction\". 2017.", "claims_and_evidence": "The authors provide theoretical justification (Proposition 2.1) showing that their state modelling objective equals the DecPOMDP objective. They perform extensive experiments on three benchmark environments (MPE, LBF, and RWARE) against multiple baselines with results that agree with their claims.", "methods_and_evaluation_criteria": "The evaluation metric (unnormalized average episodic reward) is common. The authors report results with confidence intervals averaged over six random seeds, which is sufficient for statistical significance. I appreciate not doing mean and standard error. The authors also properly analyze their method through ablation studies to verify the contribution of each component.", "theoretical_claims": "The authors refer the proof to Proposition 2.1 as “missing proof”. I am not sure what that entails. The proof is sound and follows from the fact that the set of policies encompassed by the state modelling framework includes all policies that could solve the original problem. However, I am not familiar with this methodology.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound. The authors use established benchmarks with appropriate configurations and metrics. The ablation studies are well-designed to isolate the contributions of different components.\n\nOne concern I have is that some figures show inconclusive results. A more detailed analysis in the style of rliable [1] could be more informative than some subfigures in Fig 3 and 4. \nOne minor concern is that the evaluation in MPE Spread with increasing number of agents (3, 4, 5, 8) doesn't fully demonstrate the scalability of the approach with even larger numbers of agents.\n\n[1] Agarwal et al, Deep Reinforcement Learning at the Edge of the Statistical Precipice", "relation_to_broader_scientific_literature": "The state modelling framework extends previous agent modelling approaches by learning representations with respect to policy optimization rather than as an auxiliary task. This addresses a limitation in previous works such as \"Agent modelling under partial observability for deep reinforcement learning\" (Papoudakis et al., 2021) where the aim is to learn a relationship between the trajectory of the controlled agent and the modelled agent. The paper also suggests the use of adversarial methods. \n\nThe use of AM filters to handle redundant state information is motivated by findings in \"Efficient multi-agent communication via self-supervised information aggregation\" (Guan et al., 2022) where the authors aggregate the information through an attention mechanism. Note that the authors mention the same work twice for no reason as two separate entries.\n\nThe adversarial exploration schema uses hashing as done in \"# exploration: A study of count-based exploration for deep reinforcement learning\" (Tang et al., 2017), but applies it in a novel way to the multi-agent setting. I really like the paper’s study of the smoothness of the intrinsic reward smoothness rate.", "essential_references_not_discussed": "The paper appears to cover the most relevant related works in MARL, agent modelling, and exploration. However, it could benefit from discussing:\n\n1. More recent work on belief representation learning in MARL, such as approaches using transformers or other sequence modeling techniques to handle partial observability.\n\n2. More extensive comparison with world models or model-based MARL approaches", "other_strengths_and_weaknesses": "The paper is well-written with consistent notations and the clearly stated questions throughout the main body make the paper easy to follow. The formulas are clear and the notation is consistent. Figure 1 is not entirely clear and the arrow convention is not sufficient to distinguish the gradient and the data flow. This work could benefit a lot from more justification on how the baselines are selected, especially the ones that are fairly simpler than the proposed method. \n\nThe paper has a clear ablation study but I would like to see more details on how a method with this many components can scale up to more complex settings.", "other_comments_or_suggestions": "1. Check Figure 1 and more scalability analysis. Figure 3 and 4 have inconclusive results in some subfigures, it is very clear which ones.\n\n2. Line 262 first column has a typo: “By doing do”\n\n3. The Guan et al. paper is mentioned twice in the references. There is no need or reason.\n\n4. Line 1194 should be *well-entagled* instead", "questions_for_authors": "1. How does your method scale to more agents? \n\n2. The choice of t_SNE seems somewhat arbitrary? Why choose that visualization over others? What additional observations can you make when using methods like PCA?\n\n3.  Are all the timesteps pooled together for the classification or is the accuracy reported for each timestep separately?\n\n4. Along that line, it would be very interesting if you could show more in-between steps for the visualization. The timestep 45 makes sense but it can also come across as **cherry-picked**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9I2I45a0Y", "reviewer": "Reviewer_CpzR", "summary": "This paper proposes State Modelling for Policy Enhancement through Exploration, a novel approach to cooperative multi-agent reinforcement learning in partially observable environments without communication. The method enables agents to infer meaningful belief representations about unobservable states through variational inference and self-supervised learning, while filtering out redundant information. The authors claim to enhance agents' policies both explicitly by incorporating these beliefs into policy networks and implicitly through adversarial exploration. Experiments across three benchmarks demonstrate that the proposed consistently outperforms state-of-the-art MARL algorithms, particularly in cooperative tasks that require extensive coordination and exploration.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and input, along with the positive evaluation. We respond to your comments and questions below.\n\n> 1. A more detailed analysis in the style of rliable [42] could be more informative than some subfigures in Fig 3 and 4. \n\nThanks for this comment. We will consider using rliable in the camere-ready version.\n\n> 2. Scalability with more agents\n\nSee our response in question 2. to reviewer W3Cu\n\n> 3. Comment on missing related works\n\nThank you for this comment. Does the reviewer some specific reference in mind? We will be happy to add missing related works. \n\n> 4. Figure 1 is not entirely clear and the arrow convention is not sufficient to distinguish the gradient and the data flow. \n\nThank you for this comment. Our overview of SMPE^2 and the arrow convention follows a visualization approach similar to that of many well-established MARL papers (see [43], [44], [46]). If the reviewer still has objections about the figure, we are open to discuss possible improvements. \n\n> 5. Note that the authors mention the same work twice for no reason as two separate entries. Line 262 first column has a typo: “By doing do”. The Guan et al. paper is mentioned twice in the references. Line 1194 should be well-entagled instead\n\nThanks! We will fix these minor typos.\n\n**Due to space constraints, below we include responses to some of the questions of the other reviewers.**\n\n> 6. Comparison to MAVEN\n\nWe add experimental comparison with MAVEN on challenging RWARE and LBF scenarios. We use the mean values over 6 seeds. Our method significantly outperforms this method.\n\n|Method |rware-small-4ag-hard |rware-tiny-4ag-hard|lbf:4s-11x11-3p-2f| \n|-----------------:|---------:|--------:|----------:|\n| SMPE| **6.3**|**20.1**|**98.3**| \n| MAVEN| 1.3|7.8|0.9| \n\n> 7. Real contributions of the paper\n\nRegarding the main contributions, they have been extensively highlighted:\n- in the introduction (Lines 94 left - 69 right)\n- in a series of research questions (Q1-Q8) we address\n- in important results (e.g., see lines 320-326 right: these tasks have been viewed as open challenges by [@]) \n- in section 3\n\nBased on the above, the main technical contributions of our paper are the following: We propose the state modelling optimization framework, on top of which we propose the SMPE MARL method. Based on state modelling, each agent of SMPE aims at learning meaningful beliefs about the unobserved global state, from each own perspective, by trying to predict what the other agents observe, and use her belief in order to enhance her own individual policy. To ensure that the beliefs of each agent are informative to her decision making, the framework entails learning of the belief with respect to her policy optimization. Moreover, SMPE introduces the AM filters which aim to filter out redundant global state information which may be detrimental to the inference of the beliefs, and thus of the MARL performance. Our approach introduces two novel loss objectives for learning the beliefs and the AM filters of each agent: (a) a reconstruction loss for learning the AM filters via a self-supervised manner, which ensures that the AM filters identify global state features that can be inferred by the formed agents' belief of the agent, and (b) an RL loss that ensures that the inferred beliefs incorporate information relevant to her policy optimization through backpropagation on the AM filters. Additionally, our method further harnesses the rich state information captured by the agents' beliefs. Each agent uses a count-based intrinsic reward on her own agent belief. This simple exploration schema is proved to be of great importance for joint exploration under partial observability, by encouraging each agent to discover novel observations and also by helping other agents form better explored beliefs about the unobserved global state through the means of an interesting adversarial competition. Remarkably, our experiments validate each of the above conceptual component of our method, and also suggest that SMPE outperforms state-of-the-art methods in challenging, well-established benchmarks. \n\n> 8. Even though the weight converges to including partial non-informative, it can still find a local optimum by policy. More importantly, this suboptimality is difficult to be verified.\n\nOur method controls the contribution of L_rec and L_wcritic to the AM filters through the selection of the hyperparameters: $lr_{wcritic}$ (learning rate in L_wcritic) and $lr_{w}$ (learning rate in L_rec). In practice, we found that a good hyperparameter selection is to set $lr_{wcritic}$ 100 times smaller than $lr_{w}$ (see Table 3, appendix), so that policy optimization does not impede reconstruction. One way to verify if the weight is trained well w.r.t. policy is by looking at the cumulative reward. This is because w affects z and thus both the policy and exploration. In our results (e.g., see Fig. 5) we consider that w is near-optimal w.r.t. policy.", "claims_and_evidence": "The authors provide theoretical justification (Proposition 2.1) showing that their state modelling objective equals the DecPOMDP objective. They perform extensive experiments on three benchmark environments (MPE, LBF, and RWARE) against multiple baselines with results that agree with their claims.", "methods_and_evaluation_criteria": "The evaluation metric (unnormalized average episodic reward) is common. The authors report results with confidence intervals averaged over six random seeds, which is sufficient for statistical significance. I appreciate not doing mean and standard error. The authors also properly analyze their method through ablation studies to verify the contribution of each component.", "theoretical_claims": "The authors refer the proof to Proposition 2.1 as “missing proof”. I am not sure what that entails. The proof is sound and follows from the fact that the set of policies encompassed by the state modelling framework includes all policies that could solve the original problem. However, I am not familiar with this methodology.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound. The authors use established benchmarks with appropriate configurations and metrics. The ablation studies are well-designed to isolate the contributions of different components.\n\nOne concern I have is that some figures show inconclusive results. A more detailed analysis in the style of rliable [1] could be more informative than some subfigures in Fig 3 and 4. \nOne minor concern is that the evaluation in MPE Spread with increasing number of agents (3, 4, 5, 8) doesn't fully demonstrate the scalability of the approach with even larger numbers of agents.\n\n[1] Agarwar et al, Deep Reinforcement Learning at the Edge of the Statistical Precipice", "relation_to_broader_scientific_literature": "The state modelling framework extends previous agent modelling approaches by learning representations with respect to policy optimization rather than as an auxiliary task. This addresses a limitation in previous works such as \"Agent modelling under partial observability for deep reinforcement learning\" (Papoudakis et al., 2021) where the aim is to learn a relationship between the trajectory of the controlled agent and the modelled agent. The paper also suggests the use of adversarial methods. \nThe use of AM filters to handle redundant state information is motivated by findings in \"Efficient multi-agent communication via self-supervised information aggregation\" (Guan et al., 2022) where the authors aggregate the information through an attention mechanism. Note that the authors mention the same work twice for no reason as two separate entries.\nThe adversarial exploration schema uses hashing as done in \"# exploration: A study of count-based exploration for deep reinforcement learning\" (Tang et al., 2017), but applies it in a novel way to the multi-agent setting. I really like the paper’s study of the smoothness of the intrinsic reward smoothness rate.", "essential_references_not_discussed": "The paper appears to cover the most relevant related works in MARL, agent modelling, and exploration. However, it could benefit from discussing:\n1) More recent work on belief representation learning in MARL, such as approaches using transformers or other sequence modeling techniques to handle partial observability.\n2) More extensive comparison with world models or model-based MARL approaches", "other_strengths_and_weaknesses": "The paper is well-written with consistent notations and the clearly stated questions throughout the main body make the paper easy to follow. The formulas are clear and the notation is consistent. Figure 1 is not entirely clear and the arrow convention is not sufficient to distinguish the gradient and the data flow. This work could benefit a lot from more justification on how the baselines are selected, especially the ones that are fairly simpler than the proposed method. \n\nThe paper has a clear ablation study but I would like to see more details on how a method with this many components can scale up to more complex settings.", "other_comments_or_suggestions": "Check Figure 1 and more scalability analysis. Figure 3 and 4 have inconclusive results in some subfigures, it is very clear which ones.\n\nLine 262 first column has a typo: “By doing do”\n\nThe Guan et al. paper is mentioned twice in the reference section. Both as 9 and 10. Please fix", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1XJF2subto", "reviewer": "Reviewer_hakk", "summary": "In most Multi-Agent Reinforcement Learning (MARL) problems, agents operate under partial observability, making decisions based on their observations and beliefs rather than the full state, and a naïve integration of the full state to each agent’s observation can introduce irrelevant information, hinder exploration, and degrade performance. To address this, the paper proposes a state modelling framework that enables agents to construct meaningful beliefs about the unobserved state, optimizing their policies and improving exploration.  The authors introduce State Modelling for Policy Enhancement through Exploration (SMPE2), which consists of two components: (1) self-supervised state modelling, where an encoder-decoder predicts other agents' observations using only local information while Agent Modelling filters that remove redundant joint-state features, and (2) adversarial count-based intrinsic exploration, an intrinsic reward mechanism that uses SimHash-based novelty detection to guide exploration toward novel, high-value states. Empirical results on MPE, LBF, and RWARE show SMPE2 outperforms state-of-the-art MARL baselines, with extensive ablation studies confirming the importance of its components.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and input. Please see our responses below:\n\n> 1. Scalability with more agents\n\nThanks for this question. Along with Spread-8 and LBF 7s-20x20-5p-3f (see Fig. 2, 3), we also add results on other large LBF tasks: namely 8s-25x25-8p-5f and 7s-30x30-7p-4f. We note that these tasks have been benchmarked by the AAMAS2025 paper [1]. Below, we show that SMPE outperforms all baselines in these tasks as well.\n\n|Env |SMPE |  MAA2C |MAPPO|MASER| EOI | ATM\n|-----------------:|---------:|--------:|----------:|----------:|------:|-----:|\n| 8s-25x25-8p-5f| **0.64 +- 0.12**|0.52 +- 0.24|0.41 +- 0.23|0.01 +- 0| 0.07 +- 0.02 | 0.36 +- 0.28 |\n| 7s-30x30-7p-4f| **0.74 +- 0.02**|0.71 +- 0.02   |0.57 +- 0.03|0.01 +- 0| 0.04 +- 0.01 | 0.59 +- 0.04 |\n\n\n> 2. Computational overhead: SM and AM filters a bottleneck?\n\nThanks for this question. From the table below, we observe that even in larger settings, SM and AM filters only introduce a moderate (and completely justified) extra computational overhead, less than ATM, and far less than EOI, EMC and MASER.\n\n|Env|SMPE |MAA2C|MAPPO|MASER |EOI|ATM|EMC|\n|-----------------:|---------:|--------:|----------:|----------:|------:|-----:|-----:|\n| 25x25-8p-5f| 0d 7h |0d 2h | 0d 3h | 1d 5h| 1d 20h  | 0d 8h| 4d 12h|\n| Spread-8| 0d 9h |0d 4h|0d 5h |2d 5h |3d 9h | 0d 18h | 5d 3h|\n\n> 3. Evaluated scenarios\n\nThanks for this comment. All LBF and RWARE tasks from [41] are not difficult to solve. More specifically, the RWARE tasks from [41] are not the -hard ones. Moreover, in all LBF tasks from [41] (most of them are cooperative-competitive, that is an easier version of the problem), our method converges fast to an optimal policy. This is the main reason we selected more challenging scenarios from these benchmarks. \n\nRemarkably, [1] benchmarks all of our selected tasks and explicitly highlights as open challenges most of our LBF tasks along with MPE tasks with 5 or more agents, in all of which SMPE performs significantly better than all baselines. \n\n> 4. Why was MAT not considered? SM in transformer-based architectures?\n\nThanks for this question. Since both ATM and MAT are transformer-based algorithms, we chose to include only one of them to ensure a more diverse set of baselines, rather than over-representing algorithms from the same direction of approaches. We selected ATM over MAT because MAT is not implemented in (E)PyMARL, whereas ATM and all other baselines we consider are. This ensures that all methods are evaluated under the same protocol. Additionally, MAT has already been benchmarked in the same environments (see [1]), where it generally underperforms compared to our baselines. Notably, the authors have not highlighted MAT as a \"best\" algorithm. Regarding if SMPE would fit a transformer-based architecture, we have not evaluated it yet and we have left this direction as a future work. \n\n> 5. hyperparameter tuning, such as the number of trials, tuning methodology, whether tuning was done per scenario or per environment, and parameter sharing.\n\nThanks for this comment. We followed [41]: For hyperparameters, optimization was performed for each algorithm separately in each environment (not scenario). Then from each environment, we selected one task and optimised the hyperparameters of all algorithms in this task. We evaluated algorithms using 6 independent runs (seeds). \n\nExcept for MPE, where [41] (and also [1]) noted that parameter sharing was more detrimental, we used parameter sharing for all other tasks across all algorithms. All evaluated algorithms ran with the same configuration in parameter sharing. We selected the configuration of parameter sharing based on [1,41], and we did not run further experiments with independent/sharing policies. \n\n> 6. Validity of MAPPO results\n\nThanks for this comment. We evaluated MAPPO using the EPyMARL code and adhered to the same parameters as in [41]. We are confident that all baselines were run correctly, and we can upload all log files to a GitHub repository after the camera-ready version. As the reviewer pointed out, our MAPPO results are on par with those reported in the AAMAS 2025 paper [1], where we verified that the algorithm and environment hyperparameters were the same. In contrast, [2] does not use the (E)PyMARL library, which could lead to significant differences in the evaluation protocol. Moreover, we find it peculiar that MAPPO manages to reach good performance far before 20M steps (e.g. in 2ag-tiny-hard, 4ag-small-hard). We also conducted additional experiments with different random seeds, but we did not observe the behavior described.\n\n> 7. References Not Discussed and further suggestions\n\nThanks for this comment. We will make sure to incorporate all suggestions.\n\n[1] Papadopoulos et al. (2025) - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.\n\n[2] Mahjoub et al. (2025) - Sable: A Performant, Efficient, and Scalable Sequence Model for MARL.", "claims_and_evidence": "The paper claims that the state modelling objective is equivalent to the Dec-POMDP objective, providing a proof in Appendix D.\n\nTheir claim that state modelling improves MARL performance is supported by empirical evidence from a variety of scenarios: dense (MPE), balanced sparse (LBF), and very sparse (RWARE) reward settings. Additionally, they demonstrate that SMPE2 is flexible, as it can be applied to different MARL algorithms, where they experiment in the main paper using MAA2C as the backbone. Additional results in the appendix show SMPE2-MAPPO outperforming MAPPO as well.\nThe authors further claim that AM filters retain only informative state features, preventing redundant information from degrading performance, while adversarial exploration guides agents toward high-value states, improving cooperation and exploration. Ablation studies confirm that removing these components hinders learning and slows convergence.\n\nA major concern regarding the validity of these claims is the performance of SOTA algorithms on the suggested scenarios. To my knowledge, the only literature that uses RWARE hard scenarios is [1] which compare their work using different baselines, besides a recent paper [2] that shared the same scenarios choice, however, another study using a JAXified version of these environments [3] suggests that MAPPO performs well in the proposed RWARE settings.\n\n**Reference:**\n\n[1] Christianos et al. (2021) - Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning.\n\n[2] Papadopoulos et al. (2025) - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.\n\n[3] Mahjoub et al. (2025) - Sable: A Performant, Efficient, and Scalable Sequence Model for MARL.", "methods_and_evaluation_criteria": "The paper evaluates SMPE2 in three environments  (MPE, LBF, and RWARE) and compares it against MAA2C, COMA, MAPPO, ATM, EMC, MASER, and EOI. The selection of some baselines is well motivated, as some incorporate state modelling per agent, such as EOI, and MASER. The authors provide detailed information on computational resources, hyperparameters, and benchmark settings, ensuring reproducibility.\n\nHowever, I have a few concerns regarding the evaluation:\n\n1. Scenarios selection: The authors use customized LBF and hard RWARE scenarios to showcase exploration skills. While this approach is valid, it would be beneficial to reinforce these findings on well-established scenarios from the literature [4]. This would allow for more direct comparisons with prior works.\n2. Scalability and harder settings: Further evaluation on larger-scale environments, such as RWARE larger grid and more agents (e.g., customized environments like large-8ag), could better assess SMPE2's performance in harder exploration settings with more agents. Given that many SOTA MARL algorithms may struggle to provide any signal in such settings, testing on these variants would further reinforce confidence in SMPE2's state modelling and exploration strategies.\n3. Hyperparameter tuning details: The paper does not specify how hyperparameters were tuned, how many trials were conducted, or whether tuning was performed per scenario or per environment.\n4. Suggestion to increase the number of seeds: The experiments are conducted on six random seeds, but increasing to ten seeds would improve statistical robustness, as discussed in [5,6]. \n\n**References:**\n\n [4] Papoudakis et al. (2021) - Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks.\n\n [5] Agarwal et al. (2021) - Deep RL at the Edge of the Statistical Precipice.\n\n [6] Gorsane et al. (2022) - Standardized Performance Evaluation in Cooperative MARL.", "theoretical_claims": "The paper presents Proposition 2.1, claiming that the state modelling objective is equivalent to the Dec-POMDP objective.", "experimental_designs_or_analyses": "The experiments are well-structured, with effective ablation studies highlighting each SMPE2 component’s impact. However, the authos evaluate their method using customized scenarios and ones not commonly used in prior work, including additional well-established tasks from the same settings (MPE, LBF and Rware) and additional environments in general would provide more reliable comparisons and further validate SMPE2. Additionally, I would suggest to briefly explaining t-SNE before using it in Figure 7, as some readers may not be familiar with its purpose in the ablation study.", "relation_to_broader_scientific_literature": "The paper builds on opponent modelling approaches like LIAM, SIDE, and MAVEN, but focuses on improving state representation learning rather than directly modelling other agents. The SimHash-based intrinsic reward mechanism aligns with previous count-based exploration techniques but introduces an adversarial component, distinguishing it from prior methods. In CTDE-based MARL, SMPE2 follows the centralized training decentralized execution (CTDE) paradigm, similar to MAPPO and COMA, but incorporates self-supervised learning for belief state inference, making it a novel contribution to the field.", "essential_references_not_discussed": "Despite using the transformer-based algorithm ATM, the paper lacks discussion on other SOTA transformer-based MARL methods, such as [7], which explore sequence-based representation learning as an alternative to state modelling. Additionally, it does not reference heterogeneous-agent reinforcement learning, such as [8], which studies how agents with different capabilities and roles coordinate in MARL settings. And lastly, the paper did not mention Shared Experience Actor-Critic (SEAC) [1], which focuses on improving exploration by sharing experiences among agents.\n\n**References:**\n \n[7] Wen et al. (2022) - Multi-Agent Reinforcement Learning as a Sequence Modeling Problem.\n\n [8] Zhong et al. (2023) - Heterogeneous-Agent Reinforcement Learning", "other_strengths_and_weaknesses": "**Strengths:** The work is novel and provides thorough ablation studies that effectively address most questions regarding component choices. \n\n**Weaknesses:** The computational overhead is a concern, especially when scaling to larger agent populations with such a complex network. The authors report that SMPE2 is approximately 25× faster than MASER, 30× faster than EMC, 17× faster than EOI, and 2× faster than ATM, but this comparison was done on LBF:2s-12x12-2p-2f, which only includes two agents. It remains unclear whether this speed advantage holds as the number of agents increases.", "other_comments_or_suggestions": "The paper was clear and easy to follow, which made understanding the ideas very smooth and engaging.\n\n**Syntax Suggestions:**\n\n1. Line 44:  A comma should be added for better readability.\n2. \"Due to space constraints, ...\": This phrase appears multiple times, but it would be better to directly reference the appendix section without justifying why it's not in the main text.\n3. Line 173 (\"we aim agents\") :  This phrasing feels incorrect; rewording would improve clarity.\n\n**Plot Suggestions:**\n\n1. Legends placement: Currently, all legends are placed on top of the x-axis. It would be better to move them above the plots or slightly below for visibility.\n2. Figure 5 (left plot): The legend is cropped at the edge, slightly reducing the plot's clarity.\n3. Figure 14: The legend redundantly repeats algorithm names twice.\n\n**Clarity Suggestion:**\n\n- The paper uses the format “(number)” for both citations and equations, which can be confusing for readers when referencing them in text. It would help to distinguish them visually, perhaps by adding brackets or a different formatting style for references.", "questions_for_authors": "First of all, I would like to acknowledge the thorough ablation studies in the paper. Each time I had a question about a component choice, I later found that an ablation study had already addressed it, which was great to see.\n\nThat said, I still have a few questions:\n\n1. While the results demonstrate SMPE2’s strong performance on the tasks considered, it would be helpful to explain in more detail why certain SOTA algorithms struggle in specific scenarios. Additionally, could you provide more details on the hyperparameter tuning process: how were the parameters selected? Lastly, I strongly encourage testing on well-established scenarios from the literature (e.g., from [4]), as this would allow for direct comparisons with prior works and further reinforce SMPE2’s improvements.\n2. The reported speedup comparisons are based on LBF with two agents. Does this efficiency hold for larger-scale environments, or would state modelling and AM filters introduce bottlenecks?\n3. Given that ATM (a transformer-based algorithm) was included, why was Multi-Agent Transformer (MAT) [7] not considered? Would SMPE2’s state modelling be complementary or redundant in transformer-based architectures?\n4. SMPE2 uses parameter sharing in LBF and RWARE but not in MPE. Did you test fully independent policies across all environments, and how does removing parameter sharing affect performance? Additionally, do the baseline algorithms use parameter sharing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBsFavS7Dh", "reviewer": "Reviewer_NMyw", "summary": "The paper presents a novel approach to cooperative multi-agent reinforcement learning (MARL) under partial observability by introducing a state modelling framework combined with adversarial exploration. In this framework, each agent infers a latent belief from its local observation using a variational encoder–decoder, while learnable agent modeling filters remove redundant features to capture essential global state information. The resulting SMPE2 algorithm leverages these latent beliefs by incorporating them into the policy network and using count-based intrinsic rewards to encourage exploration of novel, high-value states, thereby enhancing coordination and overcoming sparse-reward challenges. Experimental results on benchmarks like MPE, LBF, and RWARE show that SMPE2 outperforms state-of-the-art methods, leading to faster convergence and higher episodic rewards.\n\n## Update After Rebuttal\nThanks for the authors' rebuttal. However, I still feel like this paper is short of reasonable justifications. The explanations provided by the authors stand on their own suppose with no sufficient evidence to back up. Although the authors emphasized their work is theoretically sound, I have different understanding on this point. As a result, I think of this paper is not ready for publication at this moment. I suggest the authors may link their modeling and approach to some realistic implication, which is always the aim of engineering.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Motivation and Discussion about Assumptions of related work\n\nWe respectfully disagree with the reviewer's comment. In the introduction (Lines 43–64), we provide a detailed discussion of significant drawbacks and problematic assumptions of existing agent modelling (AM) methods. Remarkably, many of these claims (e.g., redundant state information, AM without exploration and policy optimization, single-agent AM) are empirically validated by our results and the extensive ablation study, demonstrating their detrimental impact on MARL performance. These challenges highlight the need for a more principled approach to AM in MARL. Regarding Lines 59–64, we believe that the assumptions outlined — namely, a priori knowledge of state features, centralized execution, and a focus solely on team games — are quite restrictive. These constraints limit the practicality of such algorithms, making them less applicable to a wide range of real-world scenarios.\n\n> 2. Assumption: z_i contains meaningful information about the unobserved state, informative for optimizing the policy. \n\nAssuming that there exists a latent belief space which contains unobserved state information meaningful for optimizing the policy has been very standard in AM (e.g., see [40, 42, 44]). We note that this  is the main assumption in order for AM to be applicable under partial observability.\n\n> 3. It rules out information that may be beneficial to decision making, but not acquired due to the defined beliefs. \n\nThe reviewer must have misunderstood that according to our definition if a state feature is beneficial to decision making, then the framework considers this feature to be \"informative\" (see first bullet, line 139). Such a feature could be identified by w_i, and thus z_i, being conditioned on w_i, is informative of this feature during exploration. Also, our definition implies that if a feature cannot be inferred through z_i and it is not relevant to policy optimization, then this feature is \"not-informative\". \n\n> 4. Comment on \"even when utilizing a compressed embedding\"\n\nThanks for this comment. It is known from [9] that providing full state information as an additional input to the policy can degrade performance due to redundant, non-informative state features. In our work, we demonstrate that this issue can persist even when using a compressed embedding z_i, as shown in Fig. 5 (left), where applying filters leads to improved performance. We will ensure that this detail is clarified in the final version.\n\n> 5. Even though the weight converges to including partial non-informative, it can still find a local optimum by policy. More importantly, this suboptimality is difficult to be verified.\n\nDue to space constraints, please see our response to Question 8. in the rebuttal of reviewer CpzR.\n\n> 6. Difference from LIAM, MAVEN and [3]\n\nDue to space constraints, please see our response to Question 10. in the rebuttal of reviewer W3Cu.\n\n> 7. Multiple components: difficult to understand from the theoretical perspective\n\nWe believe that all loss components have been well-explained in Sec. 3 and are well-validated by a plethora of experiments (see Fig. 5, 7, 15 - 26). More specifically, from a theoretical standpoint we give detailed intuition of each component:\n1. L_rec: 172 left - 182 right\n2. L_wcritic: 213 left - 218 left\n3. intrinsic reward: Sec. 3.2\n4. L_norm: 196 left - 200 left\n5. L_KL: 202 left - 207 left\n\nOur method does not rely on \"tricks\" but on **conceptual components** (1–3), motivated by how we think of the ideal MARL behavior against partial observability (see Lines 77-92 (left) and 98-104 (right)). The components 4–5 were introduced to ensure that the proposed method is theoretically sound (also illustrated in Fig. 7, 17) and adheres to our definition of state modeling (lines 152 & 205 left).\n\n> 8. Not easy to track the main contribution of this paper\n\nDue to space constraints, please see our response to Question 7. in the rebuttal of reviewer CpzR.\n\n> 9. Comparison to MAVEN\n\nDue to space constraints, please see our response to Question 6. in the rebuttal of reviewer CpzR.\n\n> 10. Question about \"intrinsically motivated...models\"\n\nSince each agent updates its policy independently, conditioned only on its own observations and beliefs, it is intrinsically motivated to seek novel observations o_i, thereby discovering novel z_i. However, since o_i serve as part of the target for other agents' decoders, they inadvertently increase the reconstruction loss of those agents. \n\n> 11. Question about \"To do so...z_i.\"\n\nSince z_i is a function of only o_i, and agent i is motivated to reach novel z_i, then the only way for the agent to achieve this is by discovering novel o_i that lead to novel z_i. In Appendix E.4.7 we show how we effectively handle the dynamic nature of the reward.\n\n[@] Papadopoulos et al. (AAMAS 2025) - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.", "claims_and_evidence": "### Supported Claims\n1. The authors have empirically demonstrated the effectiveness of each component of the proposed algorithm SMPE2.\n2. The authors claimed that \"The framework assumes that the joint state information can be redundant and needs to be appropriately filtered in order to be informative to agents.\" This has been partially verified by visualizing weight functions.\n3. The authors claimed that \"Intuitively, $w_{i}$ has an AM interpretation, as it represents the importance of each of other agents’ information to agent i’s state modelling.\" This has been verified by one example.\n4. The authors claimed that \"Note the importance of the AM filter $w_{i}$: (a) With it, although the target of ED grows linearly\nwith the number of agents, only features that can be inferred through $z_{i}$ remain as part of other agents’ observations in the reconstruction loss. (b) Without it, it would be challenging to infer meaningful embeddings $z_{i}$, due to non-informative joint state information.\" These two conditions have been verified by showing ablation study and visualizing the the projected $z_{i}$ by t-SNE.\n\n### Problematic Claims\n1. In introduction (Line 59-64), the authors blame that previous work drew multiple assumptions. This is not convincing to be a motivation. The main reason is that those assumptions are primarily used to estimate the boundary of methods. In other words, no assumptions does not imply the effectiveness to all scenarios, except that you can prove that rigorously in mathematics or do experimentation in all possible scenarios. Especially, I find that this paper also made several assumptions, such as the one illustrates the relation between latent space and observation (Line 181-184).\n2. The authors claimed that the belief $z_{i}$ contains meaningful information about the unobserved state, informative for optimizing the agent's own policy. This assumption is quite strong, for example, how is this guaranteed? \n3. As for the definition of conditions for non-informative joint state feature, the authors claimed that \"it cannot be inferred through $z_{i}$, in the sense that the agent cannot predict it conditioned on its own observation due to partial observability and non-stationarity.\" This condition is extremely strong, as it rules out information that may be beneficial to agent decision making, but not acquired due to the belief space the authors defined. As I haven't seen any clear and concrete definition for the belief space, this condition is vague. I suggest the authors can have more rigorous discussion here.\n4. The authors claimed that \"using the full state information as an extra input to the policy, even when utilizing a compressed embedding, may harm performance due to redundant state information non-informative to agent $i$.\" This may be due to the information loss brought by compression to get $z_{i}$, rather than the redundant state information.\n5. The authors claimed that \"Following the definition of the state modelling problem, we aim to ensure that $w_{i}$ (and thus $z_{i}$) incorporate information relevant to maximizing $V^{\\pi}$ and thus, $w_{i}$ to be capable of filtering non-informative state features irrelevant to maximizing agent’s future rewards.\" Although the weight $w_{i}$ is learnable, I am afraid it could be not effective to filter out the non-informative information. For example, even though the weight converges to including partial non-informative, I believe it can still find a local optimum by policy. More importantly, this suboptimality is difficult to be verified.", "methods_and_evaluation_criteria": "### Methods\nFrom the high-level view, the proposed method makes sense to the research problem this paper aims to solve. However, as the proposed method is constituted of multiple components, it is difficult to make a coherent understanding of the proposed method from the theoretical perspective.\n\n### Evaluation Criteria\nThis paper is different from other regular papers in MARL. It has conducted a lot of abalation studies from diverse dimensions, such as visualizing the learned features. This is the most critical benefit of this paper.", "theoretical_claims": "This paper has a simple theoretical claim. I have checked the proof and it should be correct.", "experimental_designs_or_analyses": "This paper has designed plenty of experiments, motivated by several research questions. I have checked these in detail, and I believe all these are sound. In addition, the experimental analyses also seem reasonable.", "relation_to_broader_scientific_literature": "This paper mainly investigates a long-standing problem in collaborative MARL, exploration and improving coordination in decentralized policies (or independent learning). The general framework proposed in this paper has no big difference from the previous work, for example, learning embeddings representing other agent behaviors [1,2] and using intrinsic reward to improve exploration [3].\n\n\n[1] Papoudakis, Georgios, Filippos Christianos, and Stefano Albrecht. \"Agent modelling under partial observability for deep reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 19210-19222.\n\n[2] Mahajan, Anuj, et al. \"Maven: Multi-agent variational exploration.\" Advances in neural information processing systems 32 (2019).\n\n[3] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\" International conference on machine learning. PMLR, 2017.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Other Weaknesses\nThis paper is not easy to comprehend. The main reason is that it has included too much information, with an overall description that establish a logic chain through all these components of the proposed method. Moreover, it is not easy to track the main contribution of this paper. It seems like a compound of several technical tricks.", "other_comments_or_suggestions": "I suggest the authors can re-organize the writing in a more logical way, to emphasize the most critical contribution delivered by this paper.", "questions_for_authors": "1. The authors are requested to address the concerns in **Claims And Evidence**.\n\n2. In addition, I believe MAVEN [1] is highly related to the framework proposed in this paper, except that it does not include an additional intrinsic reward term. For this reason, I would like to see the additional experiments for comparison between the proposed method and MAVEN.\n\n3. The authors claimed that \"Agent is intrinsically motivated to discover novel $o_{i}$ (which lead to novel $z_{i}$) which at the same time constitute unseen targets for the others’ reconstruction training. Therefore, these targets aim to adversarially increase the losses of other agents’ reconstruction models.\" Do the authors mean the mismatch between each agent's information due to independent update?\n\n4. The authors claimed that \"To do so, given that $z_{i}$ is solely conditioned on $o_{i}$, the agent is implicitly motivated to discover novel observations that must lead to novel $z_{i}$.\" What is the logic here? Can the authors provide more evidence to clarify this claim?\n\n\n[1] Mahajan, Anuj, et al. \"Maven: Multi-agent variational exploration.\" Advances in neural information processing systems 32 (2019).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "xu83PkpYgk", "reviewer": "Reviewer_MTEX", "summary": "The paper proposes a state modelling framework to infer meaningful beliefs about the unobserved state while filtering redundant information. It reconstructs other agents’ observations using an encoder-decoder. To overcome the sparse reward challenge, this paper proposes a adversarial count-based intrinsic exploration method to encourage the agents to visit novel states.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and input. We respond to your comments and questions below.\n\n> 1. The motivation of the proposed method is not clearly stated.\n\nWe respectfully disagree with the reviewer's comment. In Lines 28-36 (right), we clearly state our main motivation for this paper: \"we are interested in settings where agents lack explicit communication channels during execution. Such settings are of particular interest, because, while communication-based methods leverage inexpensive simulators for training, they may incur substantial computational overhead when executed in real-world environments\". Moreover, in the intro (Lines 43 right– 64 left), we provide a detailed discussion of significant drawbacks and problematic assumptions of existing agent modelling (AM) methods. Remarkably, many of these claims (e.g., redundant state information, AM without exploration and policy optimization, single-agent AM) are empirically validated by our results and the extensive ablation study, demonstrating their detrimental impact on MARL performance. These challenges highlight the need for a more principled approach to AM in MARL and further motivate our approach.\n \n> 2. Real contributions of the paper\n\nDue to space constraints, please see our response to Question 7. in the rebuttal of reviewer CpzR.\n\n> 3. This paper assumes that states contain information redundant for optimizing agents’ individual policies. Why?\n\nThis is an established assumption which has been validated by the NeurIPS paper ([48]-th reference in our paper) and the AAMAS paper [1], supported both by the curse of dimensionality and by intuitive reasoning. Specifically, in large and complex state spaces, each agent naturally prioritizes information that is more relevant or proximal to them while assigning lower importance to remote or less goal-related information.\n\n> 4. What does the “adversarial targets” mean? Do you mean the target entity in the actual environment?\n\nThe term \"targets\" has been used many times before the adversarial exploration section (e.g., see lines 155 right, 214 left, 175 right, 181 right, 185 right, 187 right, 195 right) in the paper to denote the reconstruction targets of the encoder-decoder.\n\n> 5. This paper designs an adversarial exploration method to “discover novel, high-value states while improving the discriminative abilities of others”. But the count-based intrinsic reward only can drive agents to visit novel states. Such an intrinsic reward could not serve the purpose of encouraging the exploration of high-value states.\n\nThanks for this comment. By the means of adversarial exploration, each agent is  motivated to reach novel states, while she challenges the state modelling abilities of other agents. By doing so, our exploration schema aims at improving the joint belief about the joint state, and thus the joint policy, as the policy network of each agent uses her belief as extra input. Therefore, our method is more likely to reach novel, high-value states than the original which hashes observations.\n\n> 6. The citation format is incorrect. In many figures, the legends obscure the main figures. Figure 1 could be presented in a better way. \n\nThanks for your comments. We will fix these minor typos in the camera-ready version. Regarding Fig. 1, we have thoroughly explain its components in high-level in Lines 147 (right) - 169 (left).\n\n> 7. What do the adversarial exploration and discriminative abilities mean?\n\nThanks for this question. By discriminative abilities of an agent under partial observability we mean the ability of the agent to identify meaningful information about the unobserved global state based on her local information. In other words, discriminative abilities represent how well the agent models the global state. Our approach defines the meaningful state information within the state modelling framework (see lines 129 left - 158 left).\n\nAdversarial exploration is extensively explained and motivated in Lines 246 (left) - 267 (left).\n\n> 8. The overall presentation of the paper lacks clarity\n\nWe kindly disagree with the above comment that there are major presentation issues about the paper. Reviewer hakk finds that \"*the paper was clear and easy to follow, which made understanding the ideas very smooth and engaging*\". Some of the points you found unclear are thoroughly explained earlier in the paper, so we believe there is a chance that you missed these explanations and this rendered the points, our motivation and contributions unclear. If this is the case, we hope through a more careful reading to appreciate our ideas and results more. Otherwise, if you insist that there are specific points which require better explanation from our end, please let us know to fix them.\n\n> 9. Scalability with more agents\n\nThanks for this question. See our response to reviewer hakk on question 1.\n\n[1] Li et al. From explicit communication to tacit cooperation: A novel paradigm for cooperative MARL. 2024", "claims_and_evidence": "The motivation of the proposed method is not clearly stated. It is hard to find the real contribution and innovation of the paper from the main contributions elaborated in the introduction.\n\nThis paper assumes that states contain information redundant for optimizing agents’\nindividual policies. Why make such an assumption?\n\nWhat does the “adversarial targets” mean? Do you mean the target entity in the actual environment?", "methods_and_evaluation_criteria": "The paper uses an encoder-decoder to model the other agents’ observations based on only the local observation, which seems reasonable. To improve exploration, the paper encourages diversity of the latent variable z by using count-based intrinsic rewards. \n\nThis paper designs an adversarial exploration method to “discover novel, high-value states while improving the discriminative abilities of others”. But the count-based intrinsic reward only can drive agents to visit novel states. Such an intrinsic reward could not serve the purpose of encouraging the exploration of high-value states.", "theoretical_claims": "I have checked the proof of the proposition 2.1.", "experimental_designs_or_analyses": "This paper conducted evaluations on MPE, LBF, and RWARE. Experiments show that the proposed method outperforms the baselines in terms of learning speed and final performance. The algorithm has obvious advantages in sparse reward tasks.", "relation_to_broader_scientific_literature": "This paper is linked to agent modeling and exploration in MARL.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n+ The experiments are good.\n+ The proposed method is novel.\n\nMain weaknesses: \n+ The overall presentation of the paper lacks clarity. The writing of this paper needs further improvement. I believe that the quality of the paper will be greatly improved after improving the writing.", "other_comments_or_suggestions": "+ The citation format is incorrect.\n+ In many figures, the legends obscure the main figures.\n+ “Learning wi (and thus zi) w.r.t. policy optimization” Such a statement is difficult to understand.\n+ Figure 1 could be presented in a better way. The current figure is difficult to read.", "questions_for_authors": "+ What does the adversarial exploration and discriminative abilities mean? How you improve the discriminative abilities of other agents?\n+ The method relies solely on local observations for agent modeling. Does it scale well as the number of agents increases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SMPE^2 a novel framework for cooperative multi-agent reinforcement learning (MARL) in partially observable environments without explicit communication. The core idea involves agents learning meaningful belief representations (z_i) of the unobserved global state from their local observations via variational inference and self-supervised learning. Crucially, the framework incorporates Agent Modellng (AM) filters (w_i) to dynamically weigh information from other agents, aiming to filter out redundant or non-informative features relative to the agent's own policy optimization. 

Reviewers generally acknowledged the novelty of the proposed approach, particularly the combination of state modeling with policy-relevance filtering and adversarial exploration mechanism. The paper tackles a significant challenge in MARL w.r.t. improving both individual and joint exploration. The strong empirical results across multiple challenging benchmarks, supported by thorough ablation studies, suggest the method is effective and its components are indeed contributing. The authors actively engaged with reviewer feedback during the rebuttal.
Decision: Accept (poster)