In-Context Fully Decentralized Cooperative Multi-Agent Reinforcement Learning
Authors: Chao Li
Bingkun BAO
Yang Gao
Venue: NeurIPS 2025 poster ()
Keywords: Cooperative multi-agent reinforcement learning
Fully decentralized setting
Context modeling
Primary area: reinforcement_learning
TL;DR: This paper enhances the fully decentralized cooperative multi-agent reinforcement learning from a context modeling perspective.

Abstract: In this paper, we consider fully decentralized cooperative multi-agent reinforcement learning, where each agent has access only to the states, its local actions, and the shared rewards. The absence of information about other agents' actions typically leads to the non-stationarity problem during per-agent value function updates, and the relative overgeneralization issue during value function estimation. However, existing works fail to address both issues simultaneously, as they lack the capability to model the agents' joint policy in a fully decentralized setting. To overcome this limitation, we propose a simple yet effective method named Return-Aware Context (RAC). RAC formalizes the dynamically changing task, as locally perceived by each agent, as a contextual Markov Decision Process (MDP), and addresses both non-stationarity and relative overgeneralization through return-aware context modeling. Specifically, the contextual MDP attributes the non-stationary local dynamics of each agent to switches between contexts, each corresponding to a distinct joint policy. Then, based on the assumption that the joint policy changes only between episodes, RAC distinguishes different joint policies by the training episodic return and constructs contexts using discretized episodic return values. Accordingly, RAC learns a context-based value function for each agent to address the non-stationarity issue during value function updates. For value function estimation, an individual optimistic marginal value is constructed to encourage the selection of optimal joint actions, thereby mitigating the relative overgeneralization problem. Experimentally, we evaluate RAC on various cooperative tasks (including matrix game, predator and prey, and SMAC), and its significant performance validates its effectiveness.

In-Context Fully Decentralized Cooperative Multi-Agent Reinforcement Learning
Chao Lil2
Bingkun Bao 13+ Yang G
School of Computer Science, Nanjing University of Posts and Telecommunications
2 State Key Laboratory of Tibetan Intelligence 3 Peng Cheng Laboratory School of Intelligent. Science and Technology, Nanjing University State Key Laboratory for Novel Software Technology, Nanjing University chaoli@n uut.t edu.ch, bingkunbaodn eept.edu.ar en,
edu.cn, gaoy@nju. edu. cn
Abstract
In this paper, we consider fully decentralized cooperative multi-agent reinforcement learning, where each agent has access only to the states, its local actions, and the shared rewards. The absence information about other agents' actions typically leads to the non-stationarity problem during per-agent value function updates, and the relative overgeneralization issue during value function estimation. However, existing works fail to address both issues simultaneously, they lack the capability to model the agents' joint policy in a fully decentralized setting. To overcome this limitation, we propose simple yet effective method named Return-Aware Context (RAC). RAC formalizes the dynamically changing task, as locally perceived by each agent, as a contextual Markov Decision Process (MDP), and addresses both non- stationarity and relative overgeneralization through return-aware context modeling. Specifically, the contextual MDP attributes the non con-stationary local dynamics of each agent to switches between contexts, each corresponding to distinct joint policy. Then, based on the assumption that the joint policy changes only between episodes, RAC distinguishes different joint policies by the training episodic return and constructs contexts using discretized episodic return values. Accordingly, RAC learns a context- based value function for each agent to address the non-stationarity issue during value function updates. For value function estimation, an individual optimistic marginal value is constructed to encourage the selection of optimal joint actions, thereby mitigating the relative overgeneralization problem. Experimentally, we evaluate RAC on various cooperative tasks (including matrix game, predator and prey, and SMAC), and its significant performance validates its effectiveness.
Introduction
In recent years, cooperative multi-agent reinforcement learning (MARL) has witnessed great advances in both algorithms (e.g., value decomposition [144 and multi-agent policy gradient [5-8] methods) and applications (e.g., autonomous vehicles [9). urban traffic management [10], and vaccine alloca- tion (TI.. Most of these advances heavily rely on the centralized training (often with decentralized execution) paradigm, where global information, particularly the joint actions of all agents, is available during training. However, direct access to other agents' actions is often unattainable in real-world settings, such as industrial scenarios where multiple robots from different companies may withhold action information due to privacy concerns or limited communication capabilities. In such cases. the fully decentralized learning becomes essential, where each agent learns solely from its local experiences, without access to the actions of other agents, during both training and execution.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, developing efficient coordinated policies under the fully decentralized learning paradigm is challenging, as the lack of access to other agents' actions typically leads to -on-staiionartty during per-agent value function updates and relative overgeneralization during value function estimation. In detail, since treating other agents as part of the environment, the local task dynamies perceived by each decentralized agent becomes non-stationary due to the evolving policies of other agents. This non-stationarity undermines the convergence of value function updates and is further exacerbated by experience replay [1). In addition, the value estimations of each agent's 's local actions may be biased by other agents' exploratory or sub-optimal action selections, hindering agents from selecting optimal joint actions, problem known as relative overgeneralization [3]. Consequently, fully decentralized learning often suffers from low efficiency and sub-optimal solutions, limiting efficient coordination.
Accordingly, we divide existing value-based based MARL approaches into two major categories. The first category of works 22 2 16 primarily focuses on the non-stationarity issue. These methods either directly access other agents' actions, design multi-agent importance sampling weights, fingerprints, or ideal transition probabilities to ensure stationary transitions, or adopt alternating policy updates among agents to maintain stationary policy updates. The second category [3] 4 17:22 mainly targets the relative overgeneralization issue, typically by rectifying the factored global action value function or employing optimistic or lenient value updates to facilitate the selection of optimal joint actions. Although both issues stem from the inaccessibility of other agents' actions, current fully decentralized MARL methods typically address either the non-stationarity issue the relative overgeneralization problem in isolation, due to their inability to model the agents' joint policy in a fully decentralized setting. As a result, they fall short of simultancously resolving both challenges.
To overcome this limitation, we propose a simple yet effective method named Return- Context (RAC). RAC formalizes the dynamically changing task locally perceived by each agent as a contextual Markov Decision Process (MDP) [23] and tackles non-stationarity and relative overgeneralization from context modeling perspective. Specifically, under the contextual MDP, RAC attributes each agent's non-stationary local task dynamics to switches between contexts, each corresponding to a distinct joint policy. Subsequently, to enable fully decentralized modeling of agents" joint policies, RAC leverages training episodic returns to differentiate among different joint policies, constructing context representations using discretized episodic return values. This return-waar modeling operates under the assumption that joint policy changes occur only between episodes, an assumption commonly adopted in existing MARL. methods, where all agents update their policies the end of each episode. Accordingly, RAC learns a context-based value function for each agent to address the non-stationarity problem during value function updates. For value function estimation, RAC derives an individual optimistic marginal value regarding all possible contexts for each agent. This guides agents toward selecting optimal joint actions and effectively mitigates the relative overgeneralization problem.
Empirically, we evaluate RAC against multiple baselines across various cooperative tasks, including matrix game, predator and prey, and StarCraft Multi-Agent Challenge (SMAC) (24). The experimental results show that RAC achieves superior performance in both learning efficiency and final outcomes, demonstrating its effectiveness in enhancing fully decentralized cooperative policy learning.
2
Related Work
In this section, we review current balue-based MARL methods, dividing them into two categories: those addressing the non-stationarity problem and those tackling the relative overgeneralization issue.
The first category of works addresses the non-stationarity problem by promoting stationary transitions or stabilizing policy updates. Specifically, canonical value decomposition methods such as VDN I and QMIX [2] assume direct access to agents' joint actions achieve stationary transitions. However, these methods often suffer from relative overgeneralization due to the representational limitation of the factored global action value function [25]. For independent Q-learning (IQL) (26) agents, the multi- agent importance sampling technique 44 assumes access to other agents' policies and constructs importance weights to decay obsolete data during experience replay. Multi-agent Fingerprints [14] employ other agents' training iteration numbers or exploration rates to estimate their policies, and augment per-agent local transitions with these estimates. However, such direct access to other agents' information is typically impractical. 12Q [16] avoids this limitation by constructing stationary ideal transitions by selecting the next state with the highest QSS value 277, enabling IQL. agents to learn optimal joint policies in a fully decentralized manner. In comparison to I2Q, which addresses both
2

---

non-stationarity and relative overgeneralization by shaping ideal state transitions, this work aims for novel return-aware perspective to tackle both issues. In addition, to ensure stationary policy updates, MA2QL. [15] enforces sequential policy updates among IQI. agents, where each agent updates its policy while keeping the others fixed. Although this strategy alleviates non-staiionarity, it often leads to low learning efficiency due to limited parallelism in policy updates.
The second category of works tackles the relative overgeneralization issue by rectifying the factored global action value function, or updating per-agent local value functions in an optimistic or lenient manner. Specifically, for value decomposition methods, weighted QMIX 17l places more weights on potentially optimal joint actions to exclusively recover correct value functions for them. QTRAN B and QPLEX [4) introduce additional terms to correct the discrepancy between the learned factored global action value function and the true joint ones. For fully decentralized IQL agents, distributed Q- learning 18 employs an optimistic decentralized value function, where each agent assumes that other agents always select their cooperative actions. This optimism encourages agents to identify and select local cooperative actions, thus addressing the relative overgeneralization issue. However, being highly optimistic makes distributed Q-learning vulnerable to stochasticity. Hysteretic Q-learning [19 20 avoids high optimism by utilizing two learning rates to update per-agent value functions with positive and negative temporal difference errors respectively. Lenient learning 21 22] shifts from optimistic value estimations to normal value estimations using decreasing lenience. Although promising, the neglect of non stationarity often limits these approaches from efficient policy learning in practice.
In summary, existing fully decentralized MARL methods face challenges in simultaneously addressing both non-stationarity and relative overgeneralization issues, due to their inability to model the agents' joint action or policy in a fully decentralized setting. To overcome this limitation, this work proposes to formalize per-agent local task dynamics by the contextual MDP, and then introduces return-awand context modeling to tackle both issues, thereby effectively enhancing fully decentralized learning.
3 Preliminary
In this section, we present the multi-agen task addressed by this work, and review the non-stationarity and relative overgeneralization problems in decentralized learning. along with the contextual MDP.
3.1 Multi-Agent Markov Decision Process
We consider a fully cooperative task that can be modeled as an Multi-agent Markov Decision Process (MMDP) (N, A, P. R. 7), where N = f1. ] represents the agent set and S denotes the state space. A A x A x ... x A is the joint action space and A' denotes the local action space of agent i E N. At each time step f,, each agent i observes the state 8t S and selects its local action of using its policy x' (a/]()). Based on all agents' joint action a = (a), a² a?......a?? environment transits to the next state 81+1 according to the state transition function P(s (st, a.) and provides all agents with the shared reward r, based on the reward function R(st, az). The goal of all agents is to learn the optimal joint policy πt' (Ir 22.
a.)) the
x x. that maximizes the expected cumulative return Ex, 12=00= ?'re). where is a discount factor. Additionally, we also evaluate our approach under the partially observable setting, as detailed in Sec. Fland Appendix A
We consider the fully decentralized learning, where each agent observes only the state 8t, its local action at. and the shared reward rt. From the perspective of each agent i, the perceived task can be modeled as an Markov Decision Process (MDP) (S, A3, p Rt 7) with dynamics defined below:
pi (((44118),, = - x (a, (5))((sst)) (sc, az),
R'(st, at) E (a, R( az),
where -² and a, denote the joint policy and the joint action of all other agents -i except agent i. Non-Stationarity. Eq. ① illustrates that each agent i's local task dynamics, represented by Pi and R.. depends on the joint policy x of other agents -i. As these agents continuously update their policies, the per-agent local task dynamics becomes non-stationary. Furthermore, when employing off-policy experience replay, the sampled transitions from the replay buffer can be viewed as following pi and R with respect to the average joint policy मं -' of other agents -i over the course of training. This non-stationarity disrupts the convergence guarantee of each agent i's value function updates.
3

---

Relative Overgeneralizatton. Due to the lack of access to other agents' actions, the value estimations of per-agent local cooperative actions may be biased by other agents' exploratory or sub-optimal action selections and thus lead to sub-optimal joint actions, problem known as relative overgeneralization.
Specifically, the local value function (st ai) of each decentralized agent can be regarded projection of the joint action value function (5e, oi; az a) IQL follows an average-based projection:
(iTT(s,, ai) = - (a, )
(00))(=(+ (8t, og, og,,;")
where (T( oi. a, ') is the joint action value function under a given joint policy π = (T', π " It is obvious that the average-basee based projection is easily affected by other agents' sub-optimal actions and suffers from the relative overgeneralization. In contrast, the optimistic projection is defined below:
Qioon (st; at) max o, Q" ais a,
where (*t ai d, represents the joint action value function of an optimal joint policy œ². The optimistic projection assumes that other agents -i always select their cooperative local actions, thus eliminating the impact of other agents' non-cooperation. Hysteretic Q-learning directly approximates Qiopt(s a;) by an optimistic value update. In contrast, our approach estimates it using optimistic marginal value derived from context- value function. We detail this distinction in Appendix. B
3.2 Contextual Markov Decision Process
A contextual MDP is defined as tuple (C, (c)), where C denotes the context space. S is the state space, and A is the action space. The function M (c) maps each context E C' to a specific MDP (S, A, Pe. Re, 7), thereby defining a family of MDPs that share the same state and action spaces but differ their transition dynamics and reward functions. In this work, we use the contextual MDP to model each agent's varying local task dynamics, which is determined by other agents' joint policies.
4 Methodology
In this section, we provide a detailed explanation of our method, Return-Aware Context (RAC). We begin by introducing the task formalization based on contextual MDP, and then delve into the process of modeling contexts using the training episodic return. Subsequently, for each decentralized agent, we propose to learn context-based value function and derive individual optimistic marginal value aimed at selecting the optimal joint actions. Finally, we summarize the overall learning procedure.
4.1 Task Formalization
As detailed in Sec. 3.1 from the perspective of each agent i, the perceived task can be modeled as an MDP (5, A", Pt. R²,, 7) where the state transition dynamics P and reward function R' depend on the joint policy x of the other agents -i. For all possible -- the local task experienced by agent i can be further decomposed into family of MDPs that share the same state and action spaces but differ in their transition and reward functions, each conditioned on specific -r, By associating each context e E C with a corresponding -i, we propose to formalize the local task of agent i as contextual MDP, which is defined as follows:
(C, S, A M()),, where M(c) : c (S, Ri 7),
(4)
where S' is the state space and A is agent i's local action space. The dynamics (s |8,, ot c) and (s, af, c) are explicitly conditioned on the context C, each corresponding to a unique
Within the contextual MDP formalization, when each agent operates in fully decentralized manner, the task dynamics is determined by underlying context, which is inherently tied to other agents' current joint policy. When an agent encounters different contexts at different time steps, the same state and local action may result in different next states and rewards. Consequently, the lack of context information prevents agent from fully capturing the task dynamics, leading to the non non-stationarity.
This contextual MDP formalization altributes non-stationarity to switches between contexts, and pro- vides a principled frumework to address this challenge by explicit context modeling. The context can be instantiated as either an estimate of other agents' current joint policy or real-time representation

---

Time step
Time step f
Episode I
Episode
$2
St
Sper
Se+1
St
Sc+1
a
a
Case
Case 2:
Context changes between time steps.
Context changes between episodes.
Two cases of context changes. Case I refers to the scenario where other agents update their joint policy at every time step or every few time steps. For example, in the left plot, the dynamics is determined by e1 at time step and by o at time step Case refers the scenario where agents update policies only between episodes; for instance, Ci persists throughout episode 1 and persists until the termination of episode e, as shown in the right plot. Empty and solid circles respectively represent observable and unobservable stochastic variables.
Figure 1:
of the current local task dynamics distribution. By augmenting per-agent caae-locall-acton pair with the inferred context, the resulting transitions become stationary, thereby enabling efficient learning.
The frequency of context changes determines the degree of non-stationarity. As illustrated in Fig. [ we define two cases: (1) Context changes between time steps, where other agents update their joint policy every (several) time steps, leading to sigmificant -onssaaiinnaiity; and (2) Context changes between episodes, where agents keep their policies fixed within each episode and update them only between episodes, resulting in moderate non-stationarity. The latter case is widely adopted by existing MARL methods. In this work, we take step toward explicitly modeling contexts within it.
4.2
Context Modeling and Usage
Based case (2) that agents update their policies only between episodes, propose to represent all agents' fixed joint policy within each episode by the training episodic return. Given a local episodic trajectory 7¹ (So: ob ro, 81, at, r1, 82, 8H) of agent i, its generation probability is defined as:
p(T')= = p(5o)
II (atlss))= (1c) P(t at),,, az);
(5)
where term ( represents all agents' joint policy and term ② that contains both the state transition probability P(8++1|*8 [8,, a.) and reward emission probability R(s,, at) is entirely determined by the environment. For fixed joint policy during each episode, the generated local trajectory as Monte Carlo sampling of all possible trajectories induced by this joint policy under the stochastic or deterministic environment. The associated episodic return implicitly represents the agents' joint policy in the return space. For joint policies that generate similar local trajectories for each agent, the episodic returns will provide them with nearby representations, and vice versa.
be viewed
Accordingly, we use the episodic return R(r') of per-agen local trajectory r' to estimate the agents' current joint policy
(m',, 1)), and refer to it as the context. Although each agent's local task dynamics solely depends on other agents' joint policy π -- achieving fully decentralized estimation of π - is intractable in practice and we instead turn to directly estimate using the episodic return R(r"). This enables fully decentralization and greater scalability, eliminating the need for each agent to separately estimate other agents -i' policies. We empirically demonstrate that estimating æ effectively mitigates the non-stationarity issue, as discussed in Sec.
Specifically, during training, each agent respectively calculates the episodic returns for sampled local trajectories. To avoid too many continuous values, we discretize the training episodic return into m intervals. Each interval x covers range of Rumas Rmii K. Ruaa Ruain (N + 1)), where Rmax and Rmin respectively denote the maximum and minimum episodic returns. For training episodic
5

---

return R(7'), the corresponding interval index K can be calculated as follows:
R((T)
K
(6)
Pmaax Ranin
170
where [.] is the floor function and K is enforced to be an integer between 0 and m -
Context- Based Value Function. We the one-hot encoding CR of k as the context of trajectory r', and learn a value function Q' (&t, Ca, aj) for each agent i, which is conditioned on the state, the local action and the context. Based on the augmented transition (st. o', Cie Ft; S1+1), the Q' is updated by:
(0') = (e.. >D [(re + max Q'(s++1: Ge, d(41) Q' Cr. =)) (7)
where 0 denotes the parameters of Q' and we sample batches of trajectories from agent i's replay buffer D' to conduct the update. C2 enables both stationary transitions and value function updates.
Individual Optimistic Marginal Value. There are still two issues during value estimation: (1) The relative overgeneralization limits agents from identifying and selecting their local cooperative actions; and (2) Each agent can not select actions using the context-based value function within the episode, where the context based on episodic return is available only when the episode terminates. To address these two issues, we propose the individual optimistic marginal value as defnedd below:
(st,,, ag) = maxe, EC Q'(st. Cx, at);
(8)
where we shape marginal value functions of per-agent local actions using the corresponding maximum context- based value estimations across all possible contexts. Eq.
8 adheres to an optimistic belief that other agents always select their optimal cooperative actions, and thus the individual optimistic marginal value regarding per-agent local action aj can reach the optimal value max,,, Ec Q'(st. Cx. af). The individual optimistic marginal value discards the effect caused by exploratory action selections of other agents, and accordingly enables each agent to select its local cooperative action, addressing the relative overgeneralization issue and leading the optimal joint policy.
sub sub-optimal
4.3 Overall Learning Procedure
Based on the learned individual optimistic marginal value, each agent i can select its local cooperative action by arg max 6' p"(st, af) = arg max.; maxc,.e Q2((s, Cx2 az). However, accurately learning the context- basse value function necessitates a comprehensive coverage across the entire return space, which is unavailable during the early training process and leads to sub-optimal outcomes. To address this issue, we separately learn a value function (st, at) and update it by the following TD loss:
LTD(O") = (eee)))) [(r, + max (' a(+1) - Q's (sy, a)))
(9)
where o¹ denotes the parameters of Q We make each agent select actions based its (is(Se, at) to generate informative transitions during the early training process, and accordingly enable an efficient update of Q'(st,
Cx. Cx,a,) a following Eq. (7) Furthermore, we propose auxiliary supervision loss:
(om) = (e (1) [xri. (- (&g
(10)
where of (-(ss) and (-((ss)) respectively denote the Boltzmann policy with respect d'(st, af) and Q3 (8t, at), which are defined as follows:
exp(o"(s),
exp(Q' a;))
n (af(st) = Enee exp(o"
<s(aa(ss) = Entes exp(Qg(st,
(11)
The intuition behind Eq. (10) is that each agent's r' is capable of addressing both non-stationarity and relative overgeneralization, and we make the decision policy TT's imitate it. As a result, we update (st,,, at) using the loss Ls(a*) = LTD(a") + 3. (a*), where 3 is scaling factor.
As presented in Algorithm [ the learning procedure of RAC as follows. We begin by initializing Q², Q for each agent i E N. During each episode, each agent i selects its local action based on Q3 and the local episode data is stored into its replay buffer Dí For agents with e-greedy policy, we keep fixed during each episode and decrease it between episodes, which is adopted by PyMARL 24 by default. This approach enables fixed policy within each episode, and is consistent with the proposed case (2) (context changes between episodes). During training, for each agent i, we sample batches of episodes from DD' and construct the context c, for each episodic trajectory r' based on the episodic return R(r'). Finally, we calculate the loss functions Le (f0', Ls(a'). and update all components by gradient descent. The entire procedure continues until the maximum training episode is reached.

---

Algorithm 1: Return-Aware Context (RAC)
Initialize necessary hyper-parameters
foreach agent i E N do Initialize parameters 8¹ 0¹, of Q', Q/s
the maximum training episode is not reached then for Each episode do foreach time step do foreach agent i E N do Select its local action of based on Q' af)
8
Store each agent i's episode into its replay buffer D Decrease for each agent if using e-greedy policy train then
10
12
foreach agent i E N do
13
Sample batches of episodes from D
Calcualte R(r*) for each episodic trajectory 7¹
14
Construct K, Gr. based Eq. (6)
Calculate (o()'), Ls (a*) based on Eq. 7 9 10
17
Update all components with gradient descent
5
Experiment
In this section, we design experiments to answer the following two questions: (1) Can RAC benefit fully decentralized learning by addressing both non-stationarity and relative overgeneraliiaaion? (See Sec. 5.1 (2) If so, which component contributes the most to its performance gain? (See Sec. 5.2
For question (1), compare RAC against fully decentralized baselines, IQL 226] Hysteretic Q- learning [1]. and 12Q [16]. on Matrix Game, Predator and Prey, and StarCraft Multi-Agent Challenge (SMAC). For question (2), conduct ablation studies to assess the effect of each component of RAC. All results are illustrated with the median performance and the standard error across five random be found in Appendix
seeds. More details about experimental setups and
A
A²
a
-12
-12
a (7.99)
-12
55.8
2-42
7.99
(7.99)
-11
2-27.
7.99
a²
-12
0
(6.022
0.04
(6.0.1
(5.994)
-11
0.03
5.94
(5.944
0.04
5.94
Payoff matrix.
(b) Q' (s, Ca. a) and o' (s.
(c) Q2 Cn. and d²(s,, a).
Table I: The payoff matrix and value functions learned by RAC. We define 4 episodic return intervals: [- 12, 0), [o, 6), [6, 8), [8, +oo). The context-based value functions Q' Gey a) and Q" Cvss a) for all contexts are presented in (b) and (c). The individual optimistic marginal values, o' (s, a) and x² (s, a). appear in the first column. Q' Cx2 a) with unattainable is marked by (*(s)emna))
5.1 Comparison Result
Matrix Game. We begin by evaluating all algorithms on didactic matrix game shown in Tab. La where two agents must select the optimal joint action (a) a to achieve the best reward. However, each agent maintains higher value estimations of its local actions a² and a when others uniformly select actions, thereby leading to relative overgeneralization where (a² a²) and (a² a²) are preferred.
Fig. 2 shows the comparison results in the matrix game. One can observe that IQL struggle in the sub-optimal joint actions with 6 rewards, which demonstrates that the average- value projection is susceptible to the relative overgeneralization issue. In contrast, RAC maintains an optimistic value estimation and accordingly excludes the effects caused by other agents' sub-optimal actions. This enables efficient selection of the optimal joint action with the best reward 8. This also applies to

---

RAC (ours)
IQL
Hysteretic-Q
12Q
Matrik Game
Predator and Prey
5m_ 6m
40
Test Return
Test Return
Test Win Rates
RAC (ours)
0.4
IQL
Hysteretic
-0
12Q
-40
0.0
0.4
0.6
0.0
1.5
Time Steps
1e4
Time Steps
1e6
Time Steps
1e6
10m_vs vs_ 2_11m
2532
3s52
0.6
0.8
Test Win Rates
0.3
Test Win
0.2
0.4
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps
1e6
Time Steps
1e6
Time Steps
1e6
35, vs_42
35_ 35_v5_52
25, vs 1sc
1.0
Rates
Test Win Rates
0.6
0.4
0.6
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
1.0
3.5
2.0
Time Steps
1e6
Time Steps
1e6
Time Steps
1e6
Figure 2: Comparison results in the matrix game, predator and prey, and seven SMAC maps.
the Hysteretic Q-learning, which follows an optimistic value update and demonstrates efficiency in simple tasks. When facing with complex tasks, such optimistic value update heavily suffers from the overestimation and non-stationarity issues, leading to poor performance. We validate this insight in subsequent comparisons. Similarly, 12Q shapes ideal transitions, which implicitly assume that other agents select cooperative policies, and learning policies on them leads to the optimal joint actions.
Furthermore, analyze the learned context- based value functions and per-agent individual optimistic marginal values. As shown in Tab. [b]and Tab. [c for agents and 2, the context- context-based value functions can accurately approximate the rewards of all joint actions, thus validating the efficiency of episodic return-aware context in representing the agents' joint policy and action. Additionally, the individual optimistic marginal value of each agent's local action adheres to the optimistic property, and equals the highest rewards when other agents select their cooperative actions. By enforcing per-agent value function to imitate it using Eq. (10). the selection of agents' optimal joint actions achieved.
Predator and Prey. We further test all methods in the predator and prey, a partially observable task featured by relative overgeneralization. We control 8 predators to capture 8 preys, and +10 reward is issued when two predators simultaneously capture a prey otherwise -2 is emitted for solitary hunting.
As depicted in Fig. 2 suffer policies, leading to reward. While Hysteretic Q-learning succeeds learning cooperative policies to achieve +40 rewards, its poor performance suggests the overestimation and non-stationarity issues caused by the optimistic value update. In contrast, RAC maintains standard Bellman update of the context-based value function, and soicly derives optimistic value estimations regarding it to calculate the individual optimistic marginal value. Therefore, RAC exhibits notable strengths in terms of both learning efficiency and asymptotic performance. I2Q fails to learn effective policies and struggles in O reward. We hypothesis that the accurate approximation to QSS value is hardly achieved in partially observable setting, hindering I2Q from shaping ideal transitions and learning cooperative policies.
suffers from the relative overgeneralization issue, and fails to learn effective
SMAC. To validate the scalability RAC in more complex tasks, we compare against baselines on seven SMAC maps (i.e., 2x3z, 3857, 3s_ 47, 3s_vs__ vs__v,__..00 5m_vs_6n 10m_vs_ jn_vs__ 1m, and 2$_ ss_))_lsc). The results, illustrated in Fig. 2 show that Hysteretic Q-learning performs the worst across all maps due to the overestimation and non-stationarity caused by its optimistic value update. These two issues
8

---

(a) Predator and Prey
(b) Predator and Prey
(c) Predator and Prey
Text Return
20
Test Return
20
Text Return
20
-20
mm5
-20
-20
m=10
B-1.0
m=15
m 0.1
10.00.0
RAC_
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.8
1.0
Time Steps
1e6
Time Steps
1e6
Time Steps
1e6
(d) Sin 6m
(e) 5m vs_ 6m
(6) Smn vs_ 6m
Test Win Rates
0.6
5=5
10
Test Win Rates
0.6
9 0.01
0.1
8 9-10
10.0
mmw
Test Win Rates
0.6
RAC
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps
1e6
Time Steps
1e6
Time Steps
1e6
Figure 3: Ablation studies regarding m and /3 of RAC in the predator and prey and 5m_vs vs_ vs_6m.
exacerbated in complex tasks and hinder learning efficiency. In contrast, while IQL achieves com- petitive performance on most maps, RAC significantly outperforms it on 5m_vs vs_mm,,___________________ 10m vs Im, 3s5z, and 3s vs_ 4z_ demonstrating its effectiveness in addressing both non-stationarity and relative overgeneralization in fully decentralized learning. On the remaining maps (2s3z, 3s_vs_57, and 2s 2s_vs_Isc) RAC performs comparably to IQL.. We hypothesis that these maps pose fewer challenges from non-stationarity and relative overgeneralization, and thus the benefit brought by RAC is not obvious. 12Q surpasses RAC on 2s3z and 3s5z, indicating that shaping ideal transitions enhances fully decentralized cooperative policy learning. However, 12Q shows inconsistent performance, performing comparably or worse on other maps. This discrepancy may be attributed to its inability of accurately estimating QSS values under partial observability. limiting its adaptability to certain maps.
5.2 Ablation Study
To examine the impact of m and B RAC's performance, we set to 5, 10, 15 and 3 to 0.01, 0.1, 1.0, 10.0, which serve as multiple baselines. Additionally, we introduce RAC___ø", the use of Q3 is excluded, and each agent i selects actions solely based on ø². This baseline allows us to evaluate the effect of separately learning Q's- As depicted in Fig. (c), in the predator and prey task, RAC_ [C____ __Ø" performs similarly to RAC, demonstrating that Q' and Ø² can be efficiently learned across the entire return space in simple tasks. As a result, as shown in Fig. (a) and (b), increasing m leads to finer partitioning of the episodic return, enabling o' to be learned more accurately. Moreover, RAC's approach of making Q imitate o' enhances its ability to learn cooperative policies, particularly as 8 inereases. However, in more complex tasks like 5m vs_ ws_6mm,, comprehensive coverage over the entire return space is hardly achieved during early training, making it difficult to efficiently learn both [AC__od', which relies only ø² for action selection, fails learn
variant where
Q' and d² Consequently, cooperative policy, as shown by its poor performance Fig. 3 (f). In such scenario, an appropriate selection of m and [ is necessitated for balancing the TD loss (Eq. (9) and the supervision loss (Eq. (10)) of Q3. As depicted in Fig. 3 (d) and (c), RAC with m=10 and 3=0.01 yields the best performance, and these settings are adopted as the default for most SMAC maps.
Discussion
In this section, we provide complementary discussion on three aspects of RAC: (1) its performance compared with MARL methods under the centralized training with decentralized execution paradigm; (2) its flexibility with respect to major components; and (3) its computational complexity analysis.
Empirical A Assessment We compare RAC with two representative baselines, QMIX (2) and VDN [.] As shown in Fig. i in Appendix C.4 RAC significantly outperforms both QMIX and VDN on the matrix game and the predator and prey tasks, where both QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the learned factored global action value

---

On multiple SMAC maps, RAC exhibits superior performance in comparison to fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this RAC's limited ability to handle partial observability and to adequately cover the whole return space.
function [25]
Flexibility Analysis. The context-based value function Q' necessitates comprehensive coverage of the return space for efficient updates. When only Q' is learned, during the early training process, coordinated behaviors are limited, and the achieved returns tend to fall within low-value region (i.e, small values of cx). Consequently, when computing the individual optimistic marginal value ('(s), ai) = max., EC (st (st,, Cx. oj), (s,,, ai) tends to approximate Q'(st, Ge, af) associated with small Cx. which in turn corresponds to sub-optimal actions yielding low episodic returns.
For complex tasks such SMAC maps, we empirically find that separately learning (5((s,, df) and optimizing it with the TD loss and the supervision loss lead to satisfactory performance. Importantly we clarify that the inclusion of Q's is not unique choice, and other alternatives are viable. In the context of SMAC maps, the ability of Q's to aid in return-space coverage can be attributed to two main factors: (1) the competitive performance of vanilla IQL, which makes Q3 a reasonable candidate for generating informative transitions; and (2) the incorporation of additional supervision signals, such as the proposed supervision loss, through which Q' helps guide Q in identifying and selecting agents' local cooperative actions, improving its effectiveness in facilitating exploration and learning.
From perspective (1), Q's could be replaced by other MARL methods that demonstrate competitive performance. From perspective (2), its exploration capability could be further enhanced by incorpo- rating intrinsic objectives (e.g., curiosity- rewards) or alternative supervision signals derived from Q' to promote broader or more targeted exploration and facilitate cooperative action selection.
Complexity Analysis. The computational complexity of RAC comprises two main components:
(a) In terms of the context-based value function Q', its inputs include both states and contexts, and the outputs correspond to the value estimates of all possible local actions of agent i. Its input space scales linearly with the dimension (or number) of contexts, and deep neural networks can generalize well across such space. The number of its output is only [A'], where A' is agent i's local action space.
(b) In terms of the individual optimistic marginal value Øi, when we set the number of contexts m too large, the enumeration of all possible contexts typically leads to high computational complexity. A promising approach is to use sampling- based derivation free heuristic search methods, such as the Cross-Entrops Method (CEM) 28 to approximate the maxima. Specifically, we iteratively draw a batch of N₂ random context samples from a candidate distribution Dr. e.g., a Gaussian. at each iteration k. The best M < No samples (with the highest context-based value estimates) are then used to fit a new Gaussian distribution DR+11 and this process repeats K times. Although sampling-based approximation can efficiently reduce the complexity caused by exhaustive search over an enormous context space and make RAC more tractable when is too large, maintaining multiple sampling processes and performing iterative updates of the Gaussian distribution add additional complexity.
7 Conclusion
For fully decentralized cooperative MARL, this paper presents RAC to address both non-stationarity and relative overgeneralization issues in a unified framework. RAC formalizes the local task dynamics of each agent as a contextual MDP, and constructs contexts using discretized episodic return values. Consequently, RAC learns a context-based value function for each agent to enable stationary policy updates, and derives an individual optimistic marginal value to facilitate the selection of optimal joint action. Extensive experiments across various cooperative tasks demonstrate its effectiveness.
Limitation and Future Work. Despite its strengths, there are three critical limitations that warrant further exploration. First, RAC necessitates comprehensive coverage of the entire return space, and learning a separate decentralized value function performs poorly complex tasks. This limitation could be addressed by combining RAC with efficient coordinated exploration techniques. Second, RAC currently relies on manual discretization of prior episodic return bounds to construct contexts, which limits adaptability and introduces high variance. To address this issue, we intend to explore adaptive context modeling techniques and alternative context representations. Third, RAC currently focuses only on the scenario where context changes between episodes (case (2)). To address case (1), where context changes between time steps, we plan to construct contexts by modeling the local task dynamics distribution over different time intervals. We leave these directions our future work.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China (No.62506172 No.62325206, No.62192783, No. No.62532003), the Key Research and Development Program of Jiangsu Province WN..B2023016--1 the Jiangsu Science and Technology Major Project NN..BG202403) the Natural Science Foundation of Jiangsu Province (No.BK2025065) Research Start-up Foundation of Recruiting Talents of Nanjing University of Posts and Telecommuni- cations (No. No.NY225026). What's 's more, the authors greatly thank all anonymous reviewers for their valuable comments, which significantly contribute to improving the quality and clarity of this work.
and the Natural Science
References
[1] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: S.tvv:10000 70670066099 2017.
[2] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-7284, 2020.
[3] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
[4] Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning arXiv preprint arXiv:2008.0100 01062, 2020.
[5] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed information processing systems, 30, 2017.
environments. Advances in neural
[6] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[7] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 1-22244,,,,, 2022.
[8] Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneous-agee reinforcement learning. Journal of Machine Learning Research, 25(32):
1 1-67, 2024.
[9] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang. David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In Conference robot learming, pages 264-285, PMLR, 2021.
[10] Xiaoqiang Wang, Liangjun Ke, Zhimin Qiao, and Xinghua Chai. Large-scale traffic signal control using novel multiagent reinforcement learning. IEEE transactions cybernetics, 51 (1):174-187, 2020.
[11] Qianyue Hao, Wenzhen Huang, Tao Feng, Jian Yuan, and Yong Li. Gat-mf: Graph attention mean field for very large scale multi-agent reinforcement learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 685-697, 2023.
[12] Jiechuan Jiang. Kefan Su, and Zongqing Lu. Fully decentralized cooperative multi-agent reinforcement learning: A survey. arXiv preprint arXiv:2401.044 0443444 2024.
[13] Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforce- ment learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review, 27(1):1-31, 2012.

---

[14] Jakob Foerster. Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference on machine learning, pages 116-1155. PMI.R, 2017.
[15] Kefan Su, Siyuan Zhou, Jiechuan Jiang, Chuang Gan, Xiangjun Wang, and Zongqing Lu. Multi- agent alternate q-learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 1791-1799, 2024.
[16] Jiechuan Jiang and Zongqing Lu. 12q: A fully decentralized q-learning algorithm. Advances in Neural Information Processing Systems, 35:20469-20488 2022.
[17] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning, Advances neural information processing systems, 33:10199 10210, 2020.
[18] Martin Lauer and Martin Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In Proceedings of the seventeenth international conference machine learning, pages 535-542, 2000.
[19] Laëtitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-lcarning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 64-69. IEEE, 2007.
[20] Shayegan Omidshafici, Jason Pazis, Christopher Amato, Jonathan How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning, pages 2681-2690. PMLR, 2017.
[21] Liviu Panait, Keith Sullivan, and Sean Luke.
Lenient learners cooperative multiagent systems. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pages 801-803, 2006.
[22] Ermo Wei and Sean Luke. Lenient learning in independent-leumer stochastic cooperative games. Journal of Machine Learning Research, 17(84):1-42. 2016.
[23] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: 1502.0229, 2015.
[24] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:
2019.
[25] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. International Conference on Machine Learning, pages 3930-3941. PMLR, 2021.
[26] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[27] Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q (s, s') with deep deterministic dynamics gradients. In International Conference on Machine Learning, pages 2825-2835. PMLR, 2020.
[28) Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss existing limitations and future works in Sec. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide both algorithmic and experimental details in Appendix C.3 What's more, the source code is available in the supplementary materials. Together, these resources ensure the reproducibility of the experimental results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the source code of our algorithm in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/ppll wttpcccpppppppppeeeeeeeeeeeeeeee httpa///ggiieee/eeeSebiiisssssss /Co////oddiissssseeeeeee ssionolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All training details can be found in Appendix C.3 and the source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results illustrated with the median performance and standard errors over five random seeds, as depicted all figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. C.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This work fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work has no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not present such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We build our algorithm and baselines upon the PyMARL 24 framework, and have appropriately cited the corresponding paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include the codebase RAC in the supplemental material, and provide README regarding its usage.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We use LLM exclusively for polishing the writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM for what should or should not be described.
19

---

A Partially Observable Setting
Task Formalization. A partially observable cooperative multi-agent task where agents make decision in fully decentralized manner is usually formalized as a Decentralized Partially Observable Markov Decision Process (Dec-PMMDP), which denoted by tuple (N, S, A, P. R, O Z),. Dec-POMDP differs from MMDP in that each agent solely has access to its local observation rather than the state. At each time step i, each agent receives its local observation oj E Z Z based on its local observation function O' (ojo)ss (x) E O, where Z' denotes agent i's observation space. Z and O represent all agents' joint observation space and joint observation function, respectively. The partial observability limits agents from fully perceiving the states, thereby exacerbating the non non-stationarity. To deal with the partial observability challenge, each agent conditions its local policy x' (a:lT)) and local value function Q' (Ti. 04) on its local action-obserrvtion history of- (% "o: 01 aj, ................................ a.-1.
Extension to Dec-POMDP. To extend RAC Dec-POMDP, learn context-based value function Q'(Ti, Cne at) for each agent i, and update it using the loss function defined as follows:
(f of .f > [(re y ma 2+11 Q'(T++1: Ca. (4+1) Q' Cx. ,f)) (12)
and the individual optimistic marginal value is derived as follows:
o (Ti, ag) = max., EC Q'(Ti, Ga, a;).
(13)
Furthermore, we condition the separately learned value function Q' (T/. at) on each agent i's r/. The corresponding TD loss defined as follows:
(rD = E ) [(r, + y max @5(F o/ - (14)
and the auxiliary supervision loss defined below:
(au = (o rt.. I [KL (17))))))))))))
(15)
where n (-1r() and T&((-Tz)) respectively denote the Boltzmann policy with respect to of (T/; oz) and Q5(T), af), which are defined as follows:
(T), x (aflr() Evees (T)
exp(Q'(Ti. af))
(16)
ExtEA exp(Q's (ss((Q(((, at))
Empirical Validation. In Sec. 5 evaluate RAC on the predator and prey and seven SMAC maps, which are featured by the partially observable challenge. The results demonstrate that RAC yields superior performance on these tasks, validating its effectiveness in solving partially observable tasks.
B The Distinction Clarification
For each agent i, the decentralized value function (*t,, af) can regarded as a projection of the true joint action value function Q(5e af, a, a) IQL adheres to an average- based projection as below:
Qi at) = Σ a, -., क "a, (a,a)s) 1= (st.
a,")
(17)
where QT afs a, ') is the joint action value function under a given joint policy π = (T²,, n -")) It is obvious that the average-bast based projection is casily affected by other agents' sub-optimal actions and suffers from the relative overgeneralization. In contrast, the optimistic projection is defined below:
Qioppt (st, at) max,, Q* af. a,7),
(18)
where Q" (st, at, a, represents the joint action value function of an optimal joint policy n². The optimistic projection assumes that other agents -i always select their cooperative local actions, thus eliminating the impact of other agents' non-cooperation. As result, each agent i can identify and select its local cooperative action based on Qi.op' (st, af), leading to the optimal joint policy.
Hysteretic Q-learning. Hysteretic Q-learning directly approximates Qi.op' at) by which is updated based on an optimistic value update below:
(' at) o if 2 0
Q'(se. of) - (2,(s,, a:) + B00
(19)
else
20

---

Table 2: Descriptions of maps used in this work.
Name
Ally Units
Enemy Units
Type
2s. vs. Isc
Stalkers
Spine Crawler
Asymmetrie Homogeneous
2 Stalkers,
2 Stalkers,
2s3z
Symmetric & Heterogeneous
3 Zcalots
3 Zealots
3 Stalkers,
3 Stalkers,
3s5z
Symmetric & Heterogeneous
5 Zealots
5 Zealots
3s_ ___vs_4z
Stalkers
4 Zealots
Asymmetric Homogeneous
3s_vs_5z
Stalkers
5 Zealots
Asymmetric Homogeneous
5m_vs_ ps_6m
5 Marines
6 Marines
Asymmetric & Homogeneous
10m_ vs_ 11m
10 Marines
11 Marines
Asymmetrie & Homogeneous
where 3 < is a complement factor and oj denotes the temporal difference error (TD-error). is defined as follows:
6 = Fz + max Q' (s1+1, - -Q'(se, of).
4241
(20)
learning target
Although Distributed Q-learning theoretically demonstrates that ('(s), af) with the optimistic value update converges the optimal joint policy, the entire update process (Eq. (I9)) solely relies on the decentralized value function Q' (st, dt). When facing with complex tasks where multiple agents strongly influence each other, fully decentralized value update often leads to instability and poor convergence. Furthermore, Hysteretic Q-learning with function approximators (particularly the deep neural networks) is susceptible to the overestimation issue, resulting in sub-optimal solutions. The poor performance of Hysteretic Q-learning empirically validates this insight.
Return-Aware Context (RAC). In contrast, RAC decomposes the learning of Qi.opt (st, a)) into two distinct sub-processes. The first is that we learn a context-based value function Q'(st, Gx, af) The context Cx represents the agents' joint policy and thus Q' (st, Ges at) approximates the true joint action value function ((s aj a, "). as empirically demonstrated by experiments the matrix game. Furthermore, the contexts enable stationary value updates of Q'(se, Ca, at). Based on the Bellman update, Q'(s Gx. at finally converges to (st,, di a, ")
The second is that derive the individual optimistic marginal value of (st, at) based on d'(ss, max,,ec ec Q' (st, Cx. a;). Consequently, as Q'(st, Cu, af) converges to @"(s ai, a) following stationary value update, of (st, at) approaches Qhoopt(st, af).
By implementing these two sub- processes, we ensure the stationary approximation of Qi at) by d' ds,s,, az) = max., ec (st, Cg, at). comparison to Hysteretic Q-learning, the context-bar based value function of RAC is free from the non-stationarity and overestimation issues. The superior performance of RAC across various cooperative tasks further validates effectiveness in enhancing fully decentralized cooperative policy learning.
C Experimental Details
C.1
Environments
Matrix Game. Matrix game is a simple two-agent cooperative stage game where two agents must select the optimal joint action (a a,)) to receive the best reward 8, In addition to the optimal joint action (a aa)), there are two sub-optimal joint actions (a2, a²) and (a", a²) that lead to 6 rewards. For each decentralized agent, when the other agent selects the action uniformly, the sub-optimal local actions a² and 3 may be preferred over the optimal ones a', leading to the relative overgeneralization.
Predator and Prey. Predator and prey is partially observable multi-agent task involving 8 predators and preys. When a predator solely tries to capture a prey, a -2 punishment is emitted. In contrast, when predators cooperate to capture a prey simultaneously, they receive +10 reward.
SMAC. The StarCraft Multi-Agent Challenge (SMAC) serves as a widely used benchmark which set of challenging maps is provided. These maps require cooperative MARL algorithms to make
21

---

Table 3: Challenges faced by all algorithms.
Algorithm
stationarity
Relative Overgeneralizatioo
IQL
Hysteretic Q-learning
12Q
decentralized control for allied units against build-in AI enemies and achieve high win rates. this paper, we choose seven maps including 5m 6m, 10m__s 11m, 2s3z, 3s5z, 3s vs 3z_4z. 3s vs vs_5z, and 2s_vs__________________s______ Isc to evaluate our algorithm. Details regarding these maps can be found in Tab. 2
C.2 Baselines
We compare RAC with several baselines, namely IQL., Hysteretic Q-learning, and I2Q, as presented in Tab. 3
Below a brief introduction to each algorithm.
IQL. IQL learns a decentralized Q'(se, ) for each agent i, and updates it using the standard TD loss. However, since IQL ignores the actions and policies of other agents, it suffers from non-stationarity during value function updates. Moreover, the Q' (8x, of) learned by IQL is an average-based projection of the true joint action value function, leading the issue of relative overgeneralization.
Hysteretic Q-learning. In this algorithm, each agent learns an optimistic projection Qi-opt(ss. af) based on the optimistic value update paradigm (Eq. (I9). Such optimistic belief assumes that other agents always select their cooperative actions, thus eliminating the negative effects caused by other agents' sub-optimal actions and addressing the relative overgeneralization issue.
12Q. 12Q addresses both non-stationarity and relative overgeneralization by shaping ideal transitions, which are constructed by selecting next states with the highest QSS value. These ideal transitions implicitly assume that other agents follow cooperative policies. By learning from such transitions, 12Q guides IQL agents toward optimal cooperative policies.
C.3 Experimental Setups
We implement all algorithms based on the PyMARL framework. For I2Q, we adopt its official source code along with the recommended hyper-parameters to conduct experiments on both predator and prey and SMAC maps. For the matrix game, we directly report I2Q's s results from the original paper.
For IQL,, we use the original implementation provided in PyMARL. For Hysteretic Q-learning, we set 300001 for the matrix game. For the predator and prey and SMAC maps, set 3=0.01, 0.1, 0.3, 0.5, and 0.7, and run multiple seeds using 3 that yields the best performance.
For our proposed algorithm RAC, the architecture of Q3 is kept consistent with Q' of IQL. It consists of hidden layer (64 units, ReLU activation), a GRU module, and linear layer (64 units) that outputs the values of all local actions. For Q'(ss, Cx. at), provide two kinds of implementations: (1) Normal-Net. This architecture contains a hidden layer (64 units, ReLU activation), a GRU module, and linear layer (64 units). The final linear layer takes the hidden states and contexts inputs, and outputs the Q values of all local actions. (2) Hyper-Net. is comprised by a hidden layer (of 64 units, ReL.U activation), GRU module, and linear layer (of 64 units). The weights and biases of the final linear layer are generated by two separate hyper-networks, which take the context as input.
For the matrix game and predator and prey tasks, we instantiate ('t Cx, a)) using the Hyper-Net architecture. For the SMAC maps, we use the Normal-Net to implement (st,, Cxy ai). Empirically, we find these configurations work well. All other settings (e.g., learning rate, batch size) are kept consistent across all algorithms.
The syper-parameters of RAC across all tasks are provided in Tab. In particular, m denotes the number of episodic return intervals, while 3 is scaling factor balancing the TD loss and the supervision loss of Q/-- Tepsiton denotes the anneal time steps of e when e-greedy policy is used for exploration. Tmaz represents the total number of training time steps, and N is the size of the replay buffer. Nbatch represents the size of sampled batches per training. is the learning rate and
22

---

RAC (ours)
IQL
Hysteretic-Q
12Q
QMIX
VDN
Matrik Game
Predator and Prey
5m_ 6m
40
1.0
Test Return
Test Return
Win Rates
0.6
-20
12L
0.2
Hysteretic
120
-40
0.0
1.5
Time Steps
1e4
Time Steps
1e6
Time Steps
1e6
10m, vs 11m
2532
3s52
M
0.8
0.8
Test Win Rates
0.8
Win
0.4
0.4
0.4
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps
1e6
Time Steps
1e6
Time Steps
1e6
35_ 35_v5_52 52
vs 1sc
1.0
Whth WM
Win Rates
0.6
0.6
0.6
0.2
0.2
0.0
0.0
0.5
3.0
2.0
0.0
0.5
1.5
2.0
0.0
1.0
3.5
2.0
Time Steps
1e6
Time Steps
1e6
Time Steps
1e6
Figure 4: Complementary comparison results of RAC against QMIX and VDN.
Table 4: seper-parameters of RAC across all tasks.
Tasks
8
T Tepxitio
T Tmar
N Nmar
Nosich
Optimizer
matrix game
50000
predator and prey
15
50k
1000000
24 vs Ise
10
0.01
50k
2050000
2832
10
0.01
50k
2050000
5000
32
RMSprop
0.0005
0.05
3s5z
10
0.01
50k
2050000
3s. vs_ 4z
5
0.01
50k
2050000
38 5z
10
0.001
2050000
5m 6m
10
0.01
50k
2050000
10m_vs_ vs_ m
10
0.01
50k
2050000
is the discounted factor. We decrease from Emaz to Emin within T Tepeiton time steps. In addition, we utilize RMSprop technique update all networks of RAC using gradient descent.
C.4
Complementary Comparison
To further evaluate the performance gap between RAC and MARL methods under the centralized training with decentralized execution (CTDE) paradigm, compare RAC against two representative CTDE-based baselines, QMIX and VDN. The comparison results are presented in Fig.
C,5
Computational Cost
We run all experiments under five different random seeds, and plot the mean/std in all figures. The experiments are carried out on a server, which comprises a AMD EPYC 7542 32-Core Processor CPU, 504GB RAM, and 8 NVIDIA GeForce RTX 4090 D GPUs.
23


Reviews:
[{"review_id": "dDCKTt0WYE", "reviewer": "Reviewer_KFLW", "summary": "This paper, \"In-Context Fully Decentralized Cooperative Multi-Agent Reinforcement Learning,\" proposes a method called Return-Aware Context (RAC) to tackle two big problems in decentralized multi-agent reinforcement learning (MARL): non-stationarity and relative overgeneralization. Basically, when agents can't see what other agents are doing, their local world keeps changing (non-stationarity), and they might make bad joint decisions because of over-optimism or bad estimates (relative overgeneralization).\n\nRAC tries to fix this by treating each agent's local task as a contextual MDP, where \"contexts\" represent different joint policies of other agents. The clever bit is using the overall episodic return (like the total score for a game) to figure out what context they're in. This helps agents learn a context-specific value function to deal with the changing environment. They also introduce an \"individual optimistic marginal value\" to push agents towards better joint actions, hopefully avoiding that overgeneralization issue. The paper assumes that these joint policies only change *between* episodes, not during them. They tested RAC on simple games, predator-prey, and StarCraft II, showing it often works better than some other decentralized methods.", "questions": "## \n\n1. **The \"Between Episodes\" Straitjacket:** You lean heavily on the assumption that other agents' policies only change *between* episodes. But agents are constantly exploring and updating with stochastic policies, even if their *parameters* only officially change at episode end. This means their *effective* policy can drift within an episode. How would RAC handle a scenario where, for example, an agent's exploration strategy changes mid-episode, subtly shifting its local dynamics for others? Does the current single, episode-wide context become stale in such cases? It seems like this assumption significantly limits the true \"non-stationarity\" it can tackle.\n2. **Dynamic Contexts and $R_{max}$/$R_{min}$:** Your context relies on discretizing episodic returns, using $R_{max}$ and $R_{min}$. How are these bounds determined, especially at the start of training when returns might be wildly unpredictable? Are they fixed or adaptive? If they're fixed and the actual returns go far outside those initial bounds, how does RAC cope? This could mess up context assignment and value learning. Did you consider more adaptive ways to define contexts, maybe using online clustering of returns or something that doesn't rely on fixed bounds?\n3. **The** $Q_{S}^{i}$ **Dependency:** You introduced $Q_{S}^{i}$ because learning $Q^{i}$ directly \"necessitates a comprehensive coverage across the entire return space, which is unavailable during the early training process\". This sounds like $Q^{i}$ alone struggles to learn effectively. Could you elaborate more on *why* $Q^{i}$ can't learn on its own early on? And if $Q_{S}^{i}$ is so crucial for \"informative transitions,\" doesn't that suggest a fundamental weakness in $Q^{i}$'s ability to drive exploration and learning? What happens if the $\\beta$ parameter (line 241) is set poorly, hindering $Q_{S}^{i}$'s ability to \"imitate\" $\\phi^{i}$?\n4. **Beyond Decentralized Baselines:** You compare against other fully decentralized methods. While that's fair, the introduction briefly mentions centralized training with decentralized execution (CTDE) as a common paradigm. Since CTDE often achieves higher performance by using global information during training, it would be valuable to see how RAC stacks up against a well-known CTDE method (like QMIX or VDN) on one or two tasks. This would really highlight the performance cost (or benefit) of staying fully decentralized and help contextualize RAC's contributions in the broader MARL landscape.", "limitations": "The authors are upfront about two limitations, which is good:\n\n1. **Exploration:** They acknowledge that comprehensive coverage of the return space is needed, and that a separate value function ($Q_{S}^{i}$) is needed because the context-based one ($Q^{i}$) performs poorly in complex tasks without it. They suggest better exploration techniques as future work. This is a reasonable point.\n2. **Context Changes Within Episode:** They explicitly state that RAC only handles contexts changing *between* episodes, not *within* time steps. They plan to model local task dynamics over time intervals for future work. This is a major limitation for real-world application, as agent policies often evolve more frequently.\n\nOne key limitation they didn't really dwell on, but which is evident, is the **sensitivity and arbitrary nature of the episodic return discretization** (the $m$ parameter). It's a crucial part of how contexts are formed, yet its optimal value seems highly task-dependent, and there’s no clear guidance on how to set it. This could make RAC quite brittle and challenging to apply to new problems without extensive hyperparameter tuning. They should consider this as another significant area for future work, perhaps exploring adaptive or more principled context formation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- **Tries to solve two important problems at once:** Many decentralized MARL methods only focus on non-stationarity OR relative overgeneralization separately.\n- **Context from returns is neat:** Using episodic returns to define a \"context\" for other agents' behavior is a pretty smart way to get around not knowing their actions. It's a pragmatic solution for truly decentralized learning.\n- **Some good empirical results:** On several tasks, especially some SMAC maps, RAC shows noticeably better performance than other fully decentralized baselines. This suggests it can indeed help in some settings.\n\n### Weaknesses\n\n- **Big Assumption: Policies only change between episodes.** The paper explicitly states that the method assumes \"joint policy changes occur only between episodes\". While they say this is common in some MARL, it’s a massive simplification of how agents might learn or react in real time. If other agents' behaviors *do* shift mid-episode (even if their learning algorithms only update at the end), RAC's fixed context for that episode seems fundamentally unable to capture that dynamism.\n- **Reactive Context: After the fact.** The context (episodic return) is only known *after* an episode finishes. This means during an episode, agents are flying blind regarding the actual context. They rely on a separate $Q_{S}^{i}$ to guide action selection, which then tries to \"imitate\" the context-aware $\\phi^{i}$. This indirect approach feels clunky and might limit how quickly agents can adapt to subtle shifts. If the \"true\" context is always a step behind, how much can it really help with real-time non-stationarity?\n- **Fragile context learning, especially early on.** The paper admits that learning the context-based value function needs \"comprehensive coverage across the entire return space,\" which is \"unavailable during the early training process and leads to sub-optimal outcomes\". This suggests the context modeling itself might be quite brittle without enough diverse experience. The introduction of $Q_{S}^{i}$ is a band-aid for this, adding complexity rather than solving the core issue of learning robust contexts.\n- **Why not universally better?** RAC doesn't win on all SMAC maps; it's \"comparably to IQL\" on some. They hypothesize these maps are \"fewer challenges from non-stationarity and relative overgeneralization\". If that's the case, it raises questions about RAC's broader utility if its benefits are only seen when these specific problems are severe. Also, I2Q sometimes beats RAC. It suggests there isn't a clear, universal win for RAC, and its advantages are task-dependent.\n- **Hyperparameter tuning nightmare?** The number of discretization intervals ($m$) and the scaling factor ($\\beta$) for the auxiliary loss are key hyperparameters, and the ablation studies show RAC is sensitive to them. Finding the right values for these will likely be a pain for new environments, and the paper doesn't offer much guidance beyond \"appropriate selection\".", "was_revised": true, "final_justification": "My rating is now a 4 (Borderline Accept).\n\nThe authors' rebuttal was very constructive and helped resolve most of my primary concerns. The detailed explanations of their ongoing work on step-wise non-stationarity and adaptive context modeling show a clear path forward for the method's limitations. The additional experimental results, including comparisons with QMIX and VDN, provided a more complete picture of RAC's performance within the broader MARL landscape.\n\nHowever, some limitations still exist. The core paper's reliance on the \"policies change only between episodes\" assumption is a significant constraint, even if it's being addressed in future work. While the authors' new results on adaptive context modeling are promising, they are preliminary and not part of the core submission. The complexity of the hyperparameter tuning for $m$ and $\\beta$ also remains a practical challenge that wasn't fully resolved.\n\nI'm raising my score because the authors demonstrated a deep understanding of my concerns and provided strong evidence that the most critical issues are being actively researched. The work, as presented with the clarifications and promises of revision, is technically solid enough to warrant acceptance, but the remaining limitations prevent a higher score.", "rebuttal": "We begin by revisiting two key insights underlying our work. (1) We aim to mitigate non-stationarity during training (*i.e.,* in value function updates) by shaping contexts based on episodic returns, under the assumption that other agents update / change their policies only between episodes (that is, the case 2 setting, where context changes between episodes). This assumption allows us to focus exclusively on episode-wise non-stationarity. (2) During execution (*i.e.,* in value function estimation), we address the issue of relative overgeneralization, thereby enabling agents to select their cooperative actions. Together, these two insights form the foundation of our approach.\n\n**1. Response to Weakness 1 and Question 1.** \n\nWe develop RAC under the assumption that contexts change only between episodes, where agents keep their **behavior (exploration)** policies fixed within each episode and update their **evaluation** policies only between episodes. To follow this assumption, for agents with $\\epsilon$-greedy policy, we keep $\\epsilon$ fixed during each episode and decrease it between episodes, as described in Lines 244-247 of the main text.\n\nWe begin with this simplified case to creatively develop an in-context fully decentralized cooperative multi-agent reinforcement learning (MARL) framework. For the more general case where contexts (other agents' policies) change between time steps, we acknowledge that RAC, in its current form, is not equipped to handle such dynamics. Extending RAC to address this more challenging scenario is the subject of our ongoing work.\n\n**2. Response to Weakness 2.**\n\nAs stated at the outset, during execution, we focus on addressing relative overgeneralization using the per-agent individual optimistic marginal value $\\phi^{i}$. To support this, we learn a separate $Q_{\\operatorname{S}}^{i}$ to achieve a coarse coverage of the return space, thereby enhancing the learning of the context-based value function $Q^{i}$, which facilitates the computation of $\\phi^{i}(s,a^{i})=\\max_{c\\in\\mathcal{C}}Q^{i}(s,a^{i},c)$. Based on the assumption that other agents update policies only between episodes, we address **episode-wise** non-stationarity during training by learning the context-based value function. In contrast, we do not need to consider real-time, step-wise non-stationarity during execution; instead, our primary objective at execution time is to guide agents toward cooperative action selection using $\\phi^{i}$ or its imitator, $Q_{\\operatorname{S}}^{i}$.\n\n**3. Response to Weakness 3.**\n\nWe agree with your argument. For cooperative tasks of moderate complexity, RAC directly using $\\phi^{i}$ to select actions achieves superior performance, as demonstrated by RAC\\_w\\_$\\phi^{i}$ in Sec 5.2. For more complex tasks, such as SMAC, we propose learning $Q_{\\operatorname{S}}^{i}$ as a coarse means to cover the return space. A more straightforward alternative would be to enhance RAC with efficient exploration techniques, for example, by separately learning an exploratory joint policy for all agents to generate informative transitions, a strategy commonly adopted in the single‑agent RL literature. We leave this extension for future work.\n\n**4. Response to Weakness 4.**\n\nRAC is specifically designed to simultaneously address both non-stationarity and relative overgeneralization. When these two issues are relatively mild, RAC is expected to perform comparably to baselines such as IQL, as evidenced by their similar performance on certain maps. However, when these challenges become more pronounced, the performance gap between RAC and other baselines widens, as demonstrated by RAC's superior results on tasks such as the matrix game, predator and prey, 5m\\_vs\\_6m, 10m\\_vs\\_11m, 3s\\_vs\\_4z, and 3s\\_vs\\_5z.\n\nRegarding scenarios where I2Q outperforms RAC, it is important to note that I2Q and RAC address both non-stationarity and relative overgeneralization from different perspectives. Each method exhibits distinct adaptability to different tasks, and it is not guaranteed that RAC will be optimal across all tasks.\n\n**5. Response to Weakness 5.** \n\nAs noted in our **Response to Weakness 3**, a more straightforward approach to covering the return space would be to equip RAC with efficient exploration techniques, such as separately learning an exploratory joint policy for all agents. This approach could eliminate the need to tune the additional hyper-parameter $\\beta$, and we plan to investigate it in the next stage of our work. For the remaining hyper-parameter $m$, we recommend determining an approximate value through a grid‑search procedure.\n\n**6. Response to Question 2.**\n\nIn this work, we accurately determine $R_{\\operatorname{max}}$ and $R_{\\operatorname{min}}$ based on prior knowledge of the task. We have also considered incremental context modeling; that is, as the return space covered by RAC expands, the context space should be expanded accordingly. As the reviewer suggested, when employing online clustering of returns to construct contexts, it is important to ensure that the same return interval consistently maps to the same context across multiple clustering processes. We plan to further explore approaches for adaptively defining contexts in the next stage of our work.\n\n**7. Response to Question 3.**\n\n$Q^{i}(s,a^{i},c)$ is conditioned on the state $s$, agent $i$'s local action $a^{i}$, and the context $c$ instantiated as the episodic return interval. During the early training process, coordinated behaviors among agents are limited, and the achieved returns tend to fall within a low-value region (corresponding to small values of $c$). Consequently, when computing the individual optimistic marginal value $\\phi^{i}(s,a^{i})=\\max_{c\\in\\mathcal{C}}Q^{i}(s,a^{i},c)$, $\\phi^{i}(s,a^{i})$ tends to approximate $Q^{i}(s,a^{i},c)$ associated with small $c$, which in turn corresponds to sub-optimal actions yielding low episodic returns. In summary, accurately learning $Q^{i}$ necessitates comprehensive coverage of the return space. We plan to achieve this by equipping RAC with efficient exploration techniques in the next stage of our work.\n\nIn this work, the learning processes of $Q_{\\operatorname{S}}^{i}$ and $Q^{i}$ form a positive feedback loop: $Q_{\\operatorname{S}}^{i}$ generates informative transitions to ensure effective updates of $Q^{i}$, while $Q^{i}$ guides $Q_{\\operatorname{S}}^{i}$ in selecting per-agent local cooperative actions to produce more informative transitions. This mutual reinforcement facilitates efficient, fully decentralized cooperative policy learning. However, if $\\beta$ is poorly set, such mutual reinforcement may be disrupted. As depicted in Fig.~3 (e), on the 5m\\_vs\\_6m, setting $\\beta=10.0$ causes $Q_{\\operatorname{S}}^{i}$ to overly imitate the poorly trained $Q^{i}$ during the early training process, and in turn $Q_{\\operatorname{S}}^{i}$ fails to generate informative transitions. Consequently, RAC fails to learn an efficient policy and suffers from poor performance.\n\n**8. Response to Question 4.**\n\n|  | RAC  | Hysteretic Q | I2Q | IQL | QMIX | VDN |\n|-------|-------|-------|-------|-------|-------|-------|\nmatrix game (test return)\t| 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 6.0$\\pm$0.0 | 6.0$\\pm$0.0 | 6.0$\\pm$0.0 |\npredator and prey (test return) | 34.32$\\pm$1.35 | 2.48$\\pm$2.57 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 | 31.13$\\pm$1.50 |\n3s\\_vs\\_4z (test win rate) | 0.83$\\pm$0.06 | 0.69$\\pm$0.12 | 0.81$\\pm$0.12 | 0.75$\\pm$0.14 | 0.98$\\pm$0.01 | 0.98$\\pm$0.01 |\n3s\\_vs\\_5z (test win rate) | 0.58$\\pm$0.06 | 0.30$\\pm$0.18 | 0.06$\\pm$0.06 | 0.41$\\pm$0.08 | 0.79$\\pm$0.19 | 0.86$\\pm$0.02 |\n5m\\_vs\\_6m (test win rate) | 0.46$\\pm$0.04 | 0.29$\\pm$0.10 | 0.42$\\pm$0.06 | 0.39$\\pm$0.05 | 0.62$\\pm$0.03 | 0.74$\\pm$0.03 |\n2s3z (test win rate)\t| 0.73$\\pm$0.02 | 0.61$\\pm$0.13 | 0.61$\\pm$0.27 | 0.70$\\pm$0.14 | 0.97$\\pm$0.02 | 0.93$\\pm$0.05 |\n2s\\_vs\\_1sc (test win rate) | 0.97$\\pm$0.04 | 0.97$\\pm$0.03 | 0.98$\\pm$0.02 | 0.98$\\pm$0.03 | 1.0$\\pm$0.0 | 0.99$\\pm$0.01 |\n\nThanks for your valuable suggestions. We evaluate QMIX and VDN on the matrix game, predator and prey, and five SMAC maps, reporting the median asymptotic performance along with the standard error across five random seeds. The results, presented in the table above, show that RAC significant outperforms both QMIX and VDN on the matrix game and predator and prey, where QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the factored global action value function. On the five SMAC maps, RAC exhibits superior performance over other fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this to RAC's limited ability to handle partial observability and to adequately cover the entire return space. Future work will focus on enhancing RAC by incorporating belief state representations and integrating more efficient exploration techniques. We will add the comparison results in the revision for a better presentation.\n\n**9. Response to Limitation.**\n\nThanks for your valuable advice. We will place emphasis on the following directions in our future work: (1) developing adaptive context modeling, (2) ensuring comprehensive coverage of the return space through efficient exploration, and (3) extending to the more general setting in which contexts (other agents' policies) change between time steps."}, {"review_id": "1V28Xukz9b", "reviewer": "Reviewer_GYBm", "summary": "This paper investigates fully decentralized collaborative multi-agent reinforcement learning. The author proposes a new method called RAC (Return Aware Context). The core idea of RAC is to view the constantly changing task dynamics that each agent sees as switching between different \"contexts\". It uses the episode returns obtained throughout the entire turn to distinguish and represent these different contexts. Based on this context, each agent learns its own value function. RAC has also designed an optimistic value estimation approach, encouraging agents to choose actions that can facilitate optimal teamwork. The author tested RAC on several tasks (matrix games, predator\n and prey, SMAC) and the results showed that it performed better than several existing fully decentralized methods (IQL, Hysteretic Q-learning, I2Q).", "questions": "See the section \"Strengths And Weaknesses”. If the author can respond well to my question, I am willing to modify the score.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n\n- The fully decentralized learning paradigm studied here has broader applicability and practical relevance than the popular CTDE (Centralized Training with Decentralized Execution) approach. Given this setting remains under-explored, the paper's research focus is significant.\n\n- The paper is generally clear and easy to understand. The proposed method seems straightforward to implement and offers a potentially effective solution for tackling non-stationarity.\n\n## Weaknesses & Concerns:\n\n- Some claims feel too strong and could confuse readers. Specifically: Why is episodic return a suitable choice for representing \"context\"? Why non-stationarity can be (solely or partly) attributed to context switching? \n\n- Compared to I2Q and Hysteretic Q algorithms, RAC does not show significant improvement.\n\n- Consider adding some baselines for CTDE methods such as VDN and QMIX in the experiment. Although comparing different paradigms is challenging, as pointed out in the introduction, CTDE is currently the mainstream paradigm in research. It is crucial to showcase performance differences and discuss trade-offs in order to demonstrate the value and potential advantages of fully decentralized research.\n\n- Considering that it is the core motivation, we would like to see more discussions in future versions on why existing methods are difficult to simultaneously solve the problems of non stationarity and relative overgeneralization, whether from a theoretical or visual analysis perspective.\n\n- In this article, it is proposed to attribute non stationarity to context switching. The reviewer would like to know if this viewpoint is proposed in this article or if there are relevant analyses in previous literature? If it is first proposed in this article, is there any proof of this hypothesis? \n\n- The context modeling section in this article should be the most important part, but the description is not clear enough. It is recommended to add an figure to illustrate this process.\n\n- Can you explain how RAC can avoid the lazy agent problem that may exist in the multi-agent learning process. If not, can you briefly introduce how to improve the RAC method in the future to alleviate this problem?\n\n- The legend in the first row and first column of Figure 2 is redundant.", "was_revised": true, "final_justification": "The author supplemented the missing specific definitions and related experiments. \nSo my final justification is \"Borderline accept\".", "rebuttal": "**1. Response to Question 1.**\n\nThe non-stationarity arises from the evolving policies $\\pi^{-i}$ of other agents $-i$. From the perspective of each agent $i$, its local task dynamics is determined by $\\pi^{-i}$, as stated in Eq. (1) of the main text. In this work, **we associate each context $c$ with a unique $\\pi^{-i}$**, and accordingly formalize the task, as locally perceived by each agent $i$, as a contextual Markov Decision Process (CMDP). A CMDP comprises a family of MDPs that share the same state and local action spaces but differ in their state transition and reward functions, which are determined by the associated contexts. Accordingly, the non-stationarity induced by changes in other agents $-i$' policies $\\pi^{-i}$ can be attributed to context switching.\n\nFurthermore, the influence of other agents $-i$' policies $\\pi^{-i}$ on a given agent $i$ is directly reflected in the rewards it receives. In this work, we focus on case (2), where all agents keep their policies fixed within each episode and change them only between episodes. Under this setting, for a fixed joint policy $\\boldsymbol{\\pi}=(\\pi^{i}, \\pi^{-i})$, the training episodic return can be regarded as a representation of $\\boldsymbol{\\pi}$ (and, by extension, of $\\pi^{-i}$ when agent $i$'s local action or policy is known). This perspective is also adopted in related works [1]. In such case, the training episodic returns are naturally associated with specific $\\pi^{-i}$, and thus we refer to them as contexts.\n\n**2. Response to Question 2.**\n\nAs shown in Fig. 2 of the main text, RAC significantly outperforms I2Q and Hysteretic Q-learning on multiple tasks, including predator and prey, 5m\\_vs\\_6m, 10m\\_vs\\_11m, 3s\\_vs\\_4z, 3s\\_vs\\_5z, and 2s\\_vs\\_1sc. We will extend our evaluation to more benchmarks to better confirm the superior performance of RAC.\n\n**3. Response to Question 3.**\n\n|  | RAC  | Hysteretic Q-learning | I2Q | IQL | QMIX | VDN |\n|-------|-------|-------|-------|-------|-------|-------|\nmatrix game (test return)\t| 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 6.0$\\pm$0.0 | 6.0$\\pm$0.0 | 6.0$\\pm$0.0 |\npredator and prey (test return) | 34.32$\\pm$1.35 | 2.48$\\pm$2.57 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 | 31.13$\\pm$1.50 |\n3s\\_vs\\_4z (test win rate) | 0.83$\\pm$0.06 | 0.69$\\pm$0.12 | 0.81$\\pm$0.12 | 0.75$\\pm$0.14 | 0.98$\\pm$0.01 | 0.98$\\pm$0.01 |\n3s\\_vs\\_5z (test win rate) | 0.58$\\pm$0.06 | 0.30$\\pm$0.18 | 0.06$\\pm$0.06 | 0.41$\\pm$0.08 | 0.79$\\pm$0.19 | 0.86$\\pm$0.02 |\n5m\\_vs\\_6m (test win rate) | 0.46$\\pm$0.04 | 0.29$\\pm$0.10 | 0.42$\\pm$0.06 | 0.39$\\pm$0.05 | 0.62$\\pm$0.03 | 0.74$\\pm$0.03 |\n2s3z (test win rate)\t| 0.73$\\pm$0.02 | 0.61$\\pm$0.13 | 0.61$\\pm$0.27 | 0.70$\\pm$0.14 | 0.97$\\pm$0.02 | 0.93$\\pm$0.05 |\n2s\\_vs\\_1sc (test win rate) | 0.97$\\pm$0.04 | 0.97$\\pm$0.03 | 0.98$\\pm$0.02 | 0.98$\\pm$0.03 | 1.0$\\pm$0.0 | 0.99$\\pm$0.01 |\n\nThanks for your valuable suggestions. We evaluate QMIX and VDN on the matrix game, predator and prey, and five SMAC maps, reporting the median asymptotic performance along with the standard error across five random seeds. The results, presented in the table above, show that RAC significant outperforms both QMIX and VDN on the matrix game and predator and prey, where QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the factored global action value function. On the five SMAC maps, RAC exhibits superior performance over other fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this to RAC's limited ability to handle partial observability and to adequately cover the entire return space. Future work will focus on enhancing RAC by incorporating belief state representations and integrating more efficient exploration techniques. We will add the comparison results in the revision for a better presentation.\n\n**4. Response to Question 4.**\n\nWe begin by providing a theoretical insight on the relationship between non-stationarity and relative overgeneralization from a context modeling perspective. Non-stationarity arises due to the lack of access to other agents' actions or policies. If a perfect context (*i.e.*, an accurate representation of other agents' joint policies) can be obtained, the context-based value function effectively conditions the current agent’s policy on the policies of the others. In this scenario, the agent is essentially learning a best-response policy, leading to convergence toward a Nash Equilibrium. However, in cooperative games, multiple Nash equilibria may exist, and not all are desirable. Without convergence to a Pareto-optimal solution, agents may fall into suboptimal equilibria, manifesting as the relative overgeneralization problem.\n\nWe attribute the inability of existing methods to simultaneously address non-stationarity and relative overgeneralization to two main factors. \n\n(1) For methods under the centralized training with decentralized execution (CTDE) paradigm, this training framework inherently alleviates the non-stationarity but retains relative overgeneralization. This issue is evident in, for example, the representational limitations of factored global action value functions in value decomposition methods [2], or the centralized-decentralized mismatch encountered in multi-agent policy gradient methods [3]. \n\n(2) For methods under the fully decentralized learning paradigm, they typically lack the capacity to model other agents' policies (contexts) due to the absence of centralized information. As a result, they primarily focus on addressing relative overgeneralization in isolation, as seen in Distributed Q-learning, Hysteretic Q-learning, lenient learners, and related methods.\n\n**5. Response to Question 5.** \n\nTo the best of our knowledge, this work is the first to formalize the task, as locally perceived by each agent, as a CMDP, thereby attributing the non-stationarity of local task dynamics to switches between unobserved contexts. A theoretical foundation for this formulation could be established by explicitly associating the context with the joint policy of other agents. We leave this theoretical extension for future work.\n\n**6. Response to Question 6.**\n\nThanks for your valuable feedback. We will incorporate the context modeling process into a figure in the revision for more clarity.\n\n**7. Response to Question 7.**\n\nThe lazy agent issue refers to the scenario where partial agents select **non-cooperative** actions. In RAC, the individual optimistic marginal value enables agents to accurately identify and select their cooperative actions. As a result, each agent tends to select its local cooperative actions, effectively addressing the lazy agent issue.\n\n**8. Response to Question 8.** \n\nThanks for your careful reading. We will address this issue in the revision.\n\n**Reference:**\n\n[1] Qiu W, Ma X, An B, et al. Rpm: Generalizable multi-agent policies for multi-agent reinforcement learning[C]//The Eleventh International Conference on Learning Representations. 2023.\n\n[2] Rashid T, Farquhar G, Peng B, et al. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Advances in neural information processing systems, 2020, 33: 10199-10210.\n\n[3] Wang Y, Han B, Wang T, et al. Dop: Off-policy multi-agent decomposed policy gradients[C]//International conference on learning representations. 2020."}, {"review_id": "GELMDCxTPK", "reviewer": "Reviewer_Fo1D", "summary": "This paper presents a novel way to perform fully decentralized multi-agent learning based on contextual MDPs. The idea is to use the episodic trajectory return as the context for a single-agent MDP, so that such an information can help the learning agent to overcome both non-stationarity (as the context reflects the behaviour of the other agents) and the relative overgeneralization phenomenon. Sampling of the agent's action is based on an optimistic marginal estimate, so that agents can coordinate during executing, assuming that the others are acting optimally.", "questions": "**Q1:** On page 5 you claim that \"The associated episodic return implicitly represents the agents’ joint policy in the return space.\". I do not think this is entirely true: different joint policies may lead to the same return in practice: for example, think of a simple meet-in-a-grid problem, where the agents only perceive some reward if they successfully meet: all the different sub-optimal joint policies that do not solve the problem will lead to the same return. Also the opposite is true: the same joint policy may lead to totally different return estimates when using Monte-Carlo samples of it. Simply put, trajectory returns are not in a bijection with the joint policies that generate them. How does your proposed solution consider these aspects when considering the return as a proxy of the joint policy? Please discuss a bit on this aspect, as I think it is an important consideration to properly understand your solution (and an interesting point in general for the methodology you are proposing).\n\n**Q2:** In Equation (5), notation is a bit confusing: $R(s_t,\\mathbf{a}_t)$ is said to be the reward emission probability, but before in the text the same exact notation was used for the actual reward function itself. Please differentiate the two terms with different notation in order to avoid confusion.\n\n**Q3:** While it is true that the use of a marginal value estimate allows to overcome the problem of relative overgeneralization and push agent to select their optimal coordinated behaviour, this does not come without a cost: the number of contexts $c_i$ may be large (it depends on how we set $\\kappa$, but using too low of a value may hinder learning and convergence), and we need to estimate the contextual action-value functions for each context-action pair (the state is an input to the $Q$-function generally), possibly making this a very expensive operation in moderately large setting. What is the impact of such an expensive operation on the computational and wall-clock time of your proposed method? Is there anything that you can do to try and dumper the cost of estimating such a marginal value function without estimating all the data-points that perhaps I am not seeing?\n\n**Q4:** The way in which you are claiming to address efficient training in the initial phase does not sound extremely convincing to me: your use of what is basically IQL should suffer from the same issues than independent $Q$-learning would have in reaching coordinated trajectories to improve the contextual estimates no? In my understanding, the real contributor here is the supervised loss term, which is indeed pushing the agents to overcome the relative overgeneralization problem, but this does not seem to take away anything from the sample complexity arising from the next of having good estimates for the larger space of contexts. Could you please discuss this aspect in more detail?\n\n**Q5:** It would be interesting to also compare RAC to a CTDE baseline (like VDN or QMIX), at least on the larger problems like SMAC, in order to assess how good are the obtained results with respect to an algorithm which leverages centralized information at training time. Of course the proposed DTDE algorithm is not expected to outperform it, but for example seeing a narrow margin from that would still be an interesting insight.\n\n**Q6:** Are you sure that there is a relative overgeneralization phenomenon happening in the SMAC setting with which RAC is helping with, as you claim in the corresponding empirical results analysis subsection? Rewards in SMAC are usually quite aligned with good trajectories, and I am not aware of any overgeneralization happening here. If so, could you explicit a bit more where this is happening?", "limitations": "Some limitations of the proposed methodology are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The problem of DTDE is a very interesting one in multi-agent reinforcement learning literature, as it is often the case that centralized information cannot be performed, and in general to reduce the increasing computational cost that such centralized components may produce. The idea of modelling the non-stationarity perceived by each agent as a contextual MDP is good, and novel up to be best of my knowledge. The paper is easy to read and understand, the idea is exposed nicely and the narrative is coherent.\n\nHowever, I feel that some aspects of the proposed method are not sufficiently discusses or clarified, for example the increased computational requirement of RAC to compute the optimistic marginal value estimate (or the induced policies required for the KL loss term in practice). Moreover, the empirical results are not astonishing, especially on the more complex SMAC settings, thus not justifying on their own right the proposed methodology, especially if this comes at the additional costs discussed before. Please see the Questions below for a more detailed discussion.", "was_revised": true, "final_justification": "My concerns have mostly been addressed by the authors rebuttal, and I consider the paper a valid contribution in the field of decentralised learning.", "rebuttal": "**1. Response to Question 1.**\n\nOur goal is to construct a representation of the agents' joint policy. For joint policies that yield similar expected returns, their corresponding value functions are also similar, and thus their learned representations should lie close to each other in the representation space. Contexts based on episodic returns naturally satisfy this requirement, and the context-based value function produces similar outputs for similar joint policies. Specifically, such contexts **cluster** the agents' joint policies according to their returns (or associated value functions), resulting in a one-to-many mapping from contexts to joint policies, rather than a one-to-one correspondence.\n\nAs noted by the reviewer, a potential drawback of representing the agents' joint policy using episodic returns is the high variance caused by the stochastic policy and the stochastic environment. A promising approach to this problem is to perform multiple Monte Carlo rollouts to approximate the expected return of a policy more reliably. We will highlight this limitation in the revised manuscript. Further extensions about reducing the high variance and exploring alternative representations are left for future work.\n\n**2. Response to Question 2.**\n\nThank you for the careful reading. We will correct this typo in the revision.\n\n**3. Response to Question 3.**\n\nThe computational complexity of our approach contains two components: \n\n(a) In terms of the context-based value function $Q^{i}(s,a^{i},c)$, its inputs include both states and contexts, and the outputs correspond to the value estimates of all possible local actions of agent $i$. Its input space scales linearly with the dimension (number) of contexts, and deep neural networks can generalize well across such space. The number of its output is only $|A^{i}|$, where $A^{i}$ is agent $i$'s local action space.\n\n(b) In terms of the individual optimistic marginal value $\\phi^{i}(s,a^{i})=\\max_{c\\in\\mathcal{C}}Q^{i}(s,a^{i},c)$, when we set the number of contexts $m$ too large, the enumeration of all possible contexts typically leads to high computational complexity. A promising approach is to use sampling-based derivation-free heuristic search method, such as CEM [1], to find approximate maxima. In detail, we iteratively draw a batch of $N$ random context samples from a candidate distribution $D_{k}$, *e.g.,* a Gaussian, at each iteration $k$. The best $M < N$ samples (with the highest context-based value estimates) are then used to fit a new Gaussian distribution $D_{k+1}$, and this process repeats $K$ times. Such a sampling-based approximation can efficiently reduce the complexity caused by exhaustive search over a enormous context space, making the approach more tractable when $m$ is too large.\n\n**4. Response to Question 4.** \n\nIn this work, the learning processes of $Q_{\\operatorname{S}}^{i}$ and $Q^{i}$ form a positive feedback loop: $Q_{\\operatorname{S}}^{i}$ generates informative transitions to ensure effective updates of $Q^{i}$, while $Q^{i}$, in turn, guides $Q_{\\operatorname{S}}^{i}$ in selecting per-agent local cooperative actions to produce more informative transitions. This mutual reinforcement facilitates efficient, fully decentralized cooperative policy learning. However, such mutual reinforcement heavily relies on appropriately weighting the supervision loss term. As acknowledged in Sec 6 (Conclusion: Limitation and Future work), RAC necessitates a thorough coverage of the entire return space to efficiently update the context-based value function, while separately learning $Q_{\\operatorname{S}}^{i}$ and tuning the supervision loss term may perform poorly in more complex tasks. To address this limitation, integrating RAC with efficient exploration techniques is a promising direction, which we leave for future work.\n\nRegarding the context-based value function, it is trained based on the transitions generated by the separately learned $Q_{\\operatorname{S}}^{i}$. More explanations about the learning complexity of the context-based value function are detailed in the **Response to Question 3**.\n\n**5. Response to Question 5.**\n\n|  | RAC  | Hysteretic Q-learning | I2Q | IQL | QMIX | VDN |\n|-------|-------|-------|-------|-------|-------|-------|\nmatrix game (test return)\t| 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 6.0$\\pm$0.0 | 6.0$\\pm$0.0 | 6.0$\\pm$0.0 |\npredator and prey (test return) | 34.32$\\pm$1.35 | 2.48$\\pm$2.57 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 | 31.13$\\pm$1.50 |\n3s\\_vs\\_4z (test win rate) | 0.83$\\pm$0.06 | 0.69$\\pm$0.12 | 0.81$\\pm$0.12 | 0.75$\\pm$0.14 | 0.98$\\pm$0.01 | 0.98$\\pm$0.01 |\n3s\\_vs\\_5z (test win rate) | 0.58$\\pm$0.06 | 0.30$\\pm$0.18 | 0.06$\\pm$0.06 | 0.41$\\pm$0.08 | 0.79$\\pm$0.19 | 0.86$\\pm$0.02 |\n5m\\_vs\\_6m (test win rate) | 0.46$\\pm$0.04 | 0.29$\\pm$0.10 | 0.42$\\pm$0.06 | 0.39$\\pm$0.05 | 0.62$\\pm$0.03 | 0.74$\\pm$0.03 |\n2s3z (test win rate)\t| 0.73$\\pm$0.02 | 0.61$\\pm$0.13 | 0.61$\\pm$0.27 | 0.70$\\pm$0.14 | 0.97$\\pm$0.02 | 0.93$\\pm$0.05 |\n2s\\_vs\\_1sc (test win rate) | 0.97$\\pm$0.04 | 0.97$\\pm$0.03 | 0.98$\\pm$0.02 | 0.98$\\pm$0.03 | 1.0$\\pm$0.0 | 0.99$\\pm$0.01 |\n\nThanks for your valuable suggestions. We evaluate QMIX and VDN on the matrix game, predator and prey, and five SMAC maps, reporting the median asymptotic performance along with the standard error across five random seeds. The results, presented in the table above, show that RAC significant outperforms both QMIX and VDN on the matrix game and predator and prey, where QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the factored global action value function. On the five SMAC maps, RAC exhibits superior performance over other fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this to RAC's limited ability to handle partial observability and to adequately cover the entire return space. Future work will focus on enhancing RAC by incorporating belief state representations and integrating more efficient exploration techniques. We will add the comparison results in the revision for a better presentation.\n\n**6. Response to Question 6.**\n\nFor a cooperative game, relative overgeneralization refers to the scenario where sub-optimal equilibrium is preferred over the optimal ones. In the SMAC benchmark, due to challenges such as partial observability and algorithmic limitations (*e.g.,* representational limitations in value decomposition methods or over-estimation in Hysteretic Q-learning), baseline methods often converge to sub-optimal policies on certain maps, as evidenced by their poor performance. In this context, relative overgeneralization steams from both environmental and algorithmic factors, rather than solely from the reward design. We will add this discussion into the revised manuscript to improve clarity.\n\n**Reference**\n\n[1] De Boer P T, Kroese D P, Mannor S, et al. A tutorial on the cross-entropy method[J]. Annals of operations research, 2005, 134(1): 19-67."}, {"review_id": "4IYwcLmuEI", "reviewer": "Reviewer_7ZBz", "summary": "The paper identifies two fundamental challenges in fully decentralized cooperative multi-agent reinforcement learning (MARL): non-stationarity and relative overgeneralization. The proposed Return-Aware Context (RAC) method models each agent’s task as a contextual MDP, using discretized episodic returns as proxies for joint policies. This is a simple yet elegant way to distinguish joint behavior patterns in a decentralized setting.", "questions": "1. The use of discretized episodic returns as contexts, while intuitive, is heuristic and lacks theoretical backing regarding discriminability among joint policies. It would strengthen the work to include analysis (or even empirical metrics) of context separability.\n2. The paper focuses only on the case where policies change between episodes (case 2). While the authors acknowledge this, it would be useful to include results or discussion on how RAC might generalize or fail under case (1), where policy changes occur within episodes.\n3. The performance depends on manual tuning of the number of context bins ($m$) and the KL supervision weight ($\\beta$). Exploring adaptive or data-driven discretization strategies could make the method more robust and easier to apply.\n4. I think the ablation studies could be further expanded. Additional ablations, such as disabling the KL term, changing the action selection strategy, or comparing with soft context clustering, would help clarify the necessity of each component.\n5. The manuscript repeats some concepts (e.g., relative overgeneralization) multiple times. The narrative could be made more concise for better readability and emphasis on novel contributions.\n\nHowever, since my research field is not focused on multi-agent systems, I will make further reasonable evaluations based on the comments of other reviewers.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: This paper provide a clear motivation and a new method. Also, the paper clearly explains the difference between average-based value projection and the optimistic marginal value, highlighting how RAC achieves better estimation and action selection without relying on other agents’ policies or actions. The authors conduct thorough evaluations on three types of tasks (Matrix Game, Predator and Prey, and SMAC maps), comparing RAC with strong baselines such as IQL, Hysteretic Q-learning, and I2Q. The empirical results demonstrate consistent and significant performance gains. RAC requires no centralized training signals and performs well under partial observability, suggesting good applicability to large-scale or real-world settings where full observability or communication is limited.\n\nWeaknesses: \n1. The use of discretized episodic returns as contexts, while intuitive, is heuristic and lacks theoretical backing regarding discriminability among joint policies. It would strengthen the work to include analysis (or even empirical metrics) of context separability.\n2. The paper focuses only on the case where policies change between episodes (case 2). While the authors acknowledge this, it would be useful to include results or discussion on how RAC might generalize or fail under case (1), where policy changes occur within episodes.\n3. The performance depends on manual tuning of the number of context bins ($m$) and the KL supervision weight ($\\beta$). Exploring adaptive or data-driven discretization strategies could make the method more robust and easier to apply.\n4. I think the ablation studies could be further expanded. Additional ablations, such as disabling the KL term, changing the action selection strategy, or comparing with soft context clustering, would help clarify the necessity of each component.\n5. The manuscript repeats some concepts (e.g., relative overgeneralization) multiple times. The narrative could be made more concise for better readability and emphasis on novel contributions.", "was_revised": false, "final_justification": "", "rebuttal": "**1. Response to Weakness 1 and Question 1.**\n\n(1) **Intuition.** Our goal is to construct a representation of the agents' joint policy. Intuitively, joint policies that yield similar expected returns tend to have similar value functions, and their representations should be located close to one another in the representation space. Contexts based on episodic returns naturally satisfy this requirement, and the context-based value function produces similar outputs for similar joint policies. Specifically, such episodic return-based contexts **cluster** the agents' joint policies according to their returns (or associated value functions), resulting in a one-to-many mapping from contexts to joint policies.\n\n(2) **Empirical Evidence.** As shown in Tab. 1 of the main text, for the matrix game, we define 4 episodic return intervals $[-12, 0), [0, 6), [6, 8), [8, +\\infty)$ based on prior knowledge of the task. For agent $1$, the contexts are associated with distinct return ranges corresponding to different policies of another agent $2$. The context-based value function $Q^{1}$ accurately approximates the rewards of all joint actions (*i.e.,* all deterministic joint policies learned by value-based methods). The above analysis also applies to agent $2$. These results empirically demonstrate the effectiveness of episodic return-based contexts in representing the agents' joint policies. \n\n(3) **Theoretical Consideration.** The theoretical foundations for the correspondence between episodic return-based contexts and the agents' joint policies have not been explored in this work. We leave this theoretical analysis for future investigation.\n\n**2. Response to Weakness 2 and Question 2.**\n\nCase (1) considers the scenario where the context changes between time steps, *i.e.*, other agents update/change their joint policy at every (or every few) time steps. In this case, the contexts based on episodic returns fail to uniquely represent the agents' changing joint policy, leading RAC to struggle with the resulting severe non-stationarity. The underlying reason is as follows: when other agents change their joint policy within episodes, the generation probability of current agent $i$'s local trajectory $\\tau^{i}=(s_{0},a_{0}^{i},r_{0},s_{1},\\ldots,s_{H-1},a_{H-1}^{i},r_{H-1},s_{H})$ is determined by multiple joint policies of other agents. Consequently, the corresponding episodic return $R(\\tau^{i})=\\sum_{t=0}^{H-1}r_{t}$ is also determined by these multiple joint policies of other agents, and thus fails to uniquely correspond to any single one. We will add this discuss in the revision to enhance clarity.\n\n**3. Response to Weakness 3 and Question 3.**\n\nThanks for your valuable suggestions. In terms of the KL supervision weight $\\beta$, a straightforward approach to ensure comprehensive coverage of the return space is to equip RAC with efficient exploration techniques. As for the number of context bins $m$, as discussed in the **Response to Question 2 of Reviewer KFLW**, adaptive context modeling may be achieved by dynamically clustering the received returns. We plan to explore both directions in the next stage of our work.\n\nWe will add this discussion in the revision to encourage further research on in-context fully decentralized cooperative MARL.\n\n**4. Response to Weakness 4 and Question 4.**\n\nThanks for your valuable advice. At this stage, we primarily focus on evaluating the impact of the KL term. We consider two specific cases: (1) When the KL term is fully disabled, the separately learned $Q_{\\operatorname{S}}^{i}$ degenerates into standard IQL, resulting in performance equivalent to that of IQL. (2) When the learning of $Q_{\\operatorname{S}}^{i}$ is removed, RAC is constrained to directly use $\\phi^{i}$ for action selection, and we assess its performance in the matrix game, predator and prey, and the SMAC map 5m\\_vs\\_6m.\n\n|  | RAC_w_$\\phi^{i}$  | Hysteretic Q-learning | I2Q | IQL |\n|-------|-------|-------|-------|-------|\nmatrix game (test return)\t| 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 8.0$\\pm$0.0 | 6.0$\\pm$0.0 |\npredator and prey (test return) | 33.41$\\pm$1.34 | 2.48$\\pm$2.57 | 0.0$\\pm$0.0 | 0.0$\\pm$0.0 |\n5m\\_vs\\_6m (test win rate) | 0.0$\\pm$0.0 | 0.29$\\pm$0.10 | 0.42$\\pm$0.06 | 0.39$\\pm$0.05 |\n\nAs shown in the table above, RAC\\_w\\_$\\phi^{i}$ achieves superior performance in the matrix game and predator and prey tasks. This highlights the effectiveness of the individual optimistic marginal value $\\phi^{i}$ in RAC for mitigating relative overgeneralization. However, RAC\\_w\\_$\\phi^{i}$ fails to learn effective policies on 5m\\_vs\\_6m. We attribute this failure to insufficient coverage of the return space, hindering the construction of meaningful contexts based on episodic returns. A detailed explanation is provided in the **Response to Question 3 of Reviewer KFLW**. For complex tasks such as SMAC, integrating RAC with efficient exploration techniques remains a promising direction for future work. We will also make every effort to complete additional ablation studies on alternative action selection strategies and soft context clustering, and incorporate them into the revised manuscript.\n\n**5. Response to Weakness 5 and Question 5.**\n\nThanks for your valuable advice. We will adopt abbreviations in the revision to enhance readability."}]

Author rebuttal:

Meta-review: This work investigates fully decentralized training in multi-agent reinforcement learning. The proposed method formulates the environment as well as the joint policy of other agents as contexts and learns context-conditioned value function for each agent. The proposed method outperforms existing decentralized training methods across different benchmarks.

Strengths:
- The studied question, decentralized learning, is under-explored in the domain of MARL
- The proposed solution of modeling the randomness as contexts is novel
- The paper is well-written

Weaknesses:
- Computing the individual optimistic marginal value adds additional computation cost

During the rebuttal session, the authors have made additional experiments and analysis to answer the questions and address the concerns of the reviewers. All reviewers select "borderline accept" for this work. In general, this work could be accepted due to its novelty and theoretical soundness.

Key changes during the rebuttal period include:
- Inclusion of comparison results against two representative CTDE methods, namely QMIX and VDN. (Reviewers KFLW, GYBm, Fo1D)
- Clarification of RAC's episodic return-based contexts in representing the agents' joint policies and addressing episode-wise non-stationarity. (Reviewers KFLW, Fo1D, 7ZBz)
- Detailed explanation of the separate learning of individual optimistic value and discussion of viable alternatives. (Reviewers KFLW, Fo1D)
- Analysis of potential extensions to handle step-wise non-stationarity. (Reviewers KFLW, 7ZBz)
- Preliminary experimental results for RAC with adaptive context modeling (referred to as Adaptive_RAC). (Reviewers KFLW, 7ZBz)
- Clarification of representing contexts using episodic returns and attributing non-stationarity to context switches. (Reviewer GYBm)
- Discussion on why existing methods struggle to simultaneously address non-stationarity and relative overgeneralization. (Reviewer GYBm)
- Discussion of reducing computational complexity when facing a large context space and presenting accurate performance assessment caused by relative overgeneralization or sub-optimal policy learning. (Reviewer Fo1D)

In summary, **I recommend accepting this work due to the additional experiments and the novelty of the proposed method.**
Decision: Accept (poster)