Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
Authors: Yueheng Li
Guangming Xie
Zongqing Lu
Venue: ICML 2025 poster (2025)
Keywords: cooperative multi-agent reinforcement learning
off-policy multi-agent reinforcement learning
value factorization
extrapolation error
Primary area: reinforcement_learning->multiagent

Abstract: Cooperative Multi-Agent Reinforcement Learning (MARL) has become a critical tool for addressing complex real-world problems. 
However, off-policy MARL methods, which rely on joint Q-functions, face significant scalability challenges due to the exponentially growing joint action space.
In this work, we highlight a critical yet often overlooked issue: erroneous Q-target estimation, primarily caused by extrapolation error.
Our analysis reveals that this error becomes increasingly severe as the number of agents grows, leading to unique challenges in MARL due to its expansive joint action space and the decentralized execution paradigm.
To address these challenges, we propose a suite of techniques tailored for off-policy MARL, including annealed multi-step bootstrapping, averaged Q-targets, and restricted action representation. Experimental results demonstrate that these methods effectively mitigate erroneous estimations, yielding substantial performance improvements in challenging benchmarks such as SMAC, SMACv2, and Google Research Football.

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
Yueheng Li'
Guangming Xie
Zongging Lu"
Abstract
Complexity
Normalizze lized Return
Cooperative Multi-Agent Reinforcement Learn- ing (MARL) has become critical tool for ad- dressing complex real-world problems. How- ever, off-policy MARL methods, which rely on joint Q-functions, face significant scalability chal- lenges due to the exponentially growing joint action space. In this work, we highlight a crit- ical yet often overlooked issue: erroneous Q- target estimation, primarily caused by extrapo- lation error. Our analysis reveals that this error becomes increasingly severe as the number of agents grows, leading to unique challenges in MARL due to its expansive joint action space and the decentralized execution paradigm. To address these challenges, we propose a suite of techniques tailored for off-policy MARL, includ- ing annealed multi-step bootstrapping, averaged Q-targets, and restricted action representation. Ex- perimental results demonstrate that these methods effectively mitigate erroneous estimations, yield- ing substantial performance improvements in chal- lenging benchmarks such as SMAC, SMACv2, and Google Research Football.
(s,,a)
MARPO MAAIC
V(s)
VIN QHIX
MAIDS
MADEPG
Number of Agents
Number of Agents
Figure 1. Comparison hetween off-policy and on-policy MARL. methods
Off-policy MARL methods, which require the estimation of joint Q-functions, are especially affected by this exponential growth. As illustrated Fig. (left), the computational complexity of off-policy methods increases substantially with the number of agents, in contrast on-policy meth- ods, which estimate the joint state value function directly. Although recent approaches, such as value factorization methods (Son et al., 2019; Sunehag et al., 2018; Rashid et al., 2020b; Wang et al., 2021b), have shown promise in addressing this scalability issue. their performance often deteriorates as the number of agents grows. This limitation is evident in Fig. (right), which compares popular off- policy and on-policy MARL methods the MPE (Lowe et al., 2017) spread task. While both categories of methods achieve comparable performance for smaller agent popula- tions (N < 3), off-policy methods experience significant performance drop as the number of agents increases. Even value factorization techniques, though somewhat more scal- able, fail to address this degradation fully.
1. Introduction
Cooperative Multi-Agent Reinforcement Learning (MARL) has proven to be a powerful framework for tackling a di- verse range of complex real-world challenges, including autonomous driving (Zhou et al., 2020), traffic manage- ment (Singh et al., 2020), and robot swarm coordination (Hüttenrauch et al., 2019; Zhang al., 2021a). However, the inherent complexity of these scenarios introduces sig- nificant challenges, particularly with scalability, as the joint action space grows exponentially with the number of agents.
In this paper, investigate the root causes behind the limi- tations of off-policy MARL methods. Our analysis identifies critical yet often overlooked issue: estimation errors in the Temporal Difference (TD) target. These errors stem from two MARL-specific challenges: (1) the exponentially ex- panding joint action space amplifies extrapolation errors in TD target estimation, and (2) addressing these errors neces- sitates strict monotonicity in value factorization methods.
'College of Engineering, "Institut of Artificial Intelligence, "Scoool of Computer Science, Peking University. Correspon- dence to: Guangming Xie trreginng@pku.ddun Zongqing Lu <zongqing @ pku.edu.cn>.
Building on these observations, we propose suite of tech- niques to mitigate estimation errors in off-policy MARL, including annealed multi-step bootstrapping, averaged TD targets, and restricted action representations. These tech- niques improve performance by leveraging temporally ex-
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
tended trajectories to reduce bias, averaging target values to lower variance, and simplifying target estimation to al- leviate extrapolation errors. When integrated into existing off- off-policy MARL methods, these techniques yield substan- tial performance gains across a variety of challenging tasks, including SMAC (Samvelyan et al., 2019), SMACv2 (Ellis et al., 2023), and Google Research Football (GRF) (Kurach et al., 2020).
(s, a; 0) parameterized by 0. Within the framework of deep Q-learning, updates depend batch of transitions (s, a, r, r') derived from the replay buffer D. The training of the value function aims to minimize the mean square error:
I(0) B(eaan))) [(Q a; =)
(1)
where y = + 7 maxa Q(5', 0') represents the TD tar- get. The function Q(=:6') corresponds to the target network parameterized by 0'. The periodically updated o' ensures a consistent target across multiple iterations.
2. Background
2.1. Dec-POMDP and CTDE
2.3. Value Factorization
We consider Decentralized Partially Observable Markov De- cision Process POMDP) (Oliehoek & Amato, 2016) in
Value factorization methods involve learning factorized value function that encompasses per-agent utilities, denoted as [Qr(T,, a,)] and is rooted in the principles of Q- learning. A prominently discussed concept in this context is (IGM) (Son et al., 2019), designed to ensure that the locally greedily selected action aligns with the jointly optimal action. Adhering to this constraint, vari- ous value factorization methods have been proposed, with some notable examples being VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2020b), QTRAN (Son et al., 2019), and QPLEX (Wang et al.. 202 In the representation of the joint Q-function, VDN employs an additive assumption: Q a) = E QT a.). On the other hand, QMIX utilizes a monotonic mixing function
modeling cooperative multi-agent tasks. The Dec-POMDP characterized by the tuple A, )),, where N is the set of agents, S is the set of states, A is the set of actions, is the reward funetion, P is the transi- tion probability function, E is the individual partial ob- servation generated by the observation function O and is the discount factor. At each timestep, each agent d E N receives a partial observation Og E Z according to (s); at state E S. Then, each agent selects an action a, A according its action-observation history T E (Z x A)", collectively forming a joint action denoted as The state undergoes transition to the next state s' in accordance with Pss'|s,, a), and agents receive a shared reward r. The joint action-value function is expressed as (=(ss: a.) = Eg 11:1:00 notes the joint policy.
E 7're+s), where π de-
Q(s a) = fs. QI(1,, 01), ... Qu(Tn, an)) with 0s
aQ, 2 0,
(2)
This work adheres to the Centralized Training with Decen- tralized Execution (CTDE) (Olichoek et al., 2008; Krae- mer & Banerjee, 2016) paradigm. In the training phase, CTDE enables policy training to capitalize on globally avail- able information and facilitates the exchange of information among agents. Conversely, during the execution phase, each agent is restricted to accessing solely its individual observaiiservtiion history, thereby embodying decentral- ized execution principles.
where the function f is approximated using a hypereework that takes the global state s as input and produces non- negative weights, ensuring monotonieity. QPLEX extends the factorization to enable greater representational capacity via duplex advantage-weighted formulation:
Q(s, a) ( -[(((r.a)--))((((.)) EQ((ra)) (3)
where As(T, a.) Qi(T, max;; Qi(T, a)) The key to QPLEX's full expressiveness lies in the weight parameter (= > 0, which is conditioned on the joint action space. Value-based RL methods typically involve the iter- ative adjustment of Q-functions based on the Bell- man equation:
2.2. Value-Based RL
For consistency throughout the remainder of this paper, we denote individual Q-functions as Qi(s, as). even though some methods use T. or T as inputs.
+ yP= Qu. where T denotes the Bellman operator and P=Q = Ex P(s'|s, ss, a) Ea ((a')s)QU a'). Restricting the pol- icy to be greedy w.r.t the current Q-function, i.e., π E GQ)), where G(Q) is the set of all greedy policies w.r.t Q, trans- forms the operator into the Bellman optimality operator T. resulting in the Q-learning update Q++1 TQ
Qk+1 = T=Qu
3. Analysis on Off-Policy MARL
In this section, we investigate the challenges that hinder the performance of off-policy MARL algorithms. We begin with an error decomposition of the Q-function, followed by identifying two MARL-specific issues related to the estima- tion errors of the TD target.
In scenarios with a large state space, the value is of- ten estimated using a differentiable function approximator
2

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
3.1. Error Decomposition of Q-function
games (Mahajan al., 2019; Son al., 2019; Wang et al., 2021b;c; Rashid al., 2020a). This simplification turns the problem into one of Q-function updating toward the true target, which mainly involves TAE, while overlooking the impact of TEE, the second term in Eq. (4).
In practical deep Q-learning with function approximation, the learning of Q-functions may encounter various errors. Following the definition from Anschel et al. (2017), the error A between the current value and the optimal value can be decomposed into three terms:
In this paper, we identify that TEE is a crucial issue for the performance of off off-policy MARL algorithms. primar- ily driven by extrapolation error. While both TAE and TEE capture Q- Q-function estimation errors, TEE specifically reflects the accuracy of predictions for Q('', whereas TAE pertains to ((s, a). This distinction is critical because a' may not be observed in past trajectories, which can lead to significant extrapolation errors when predicting its value.
A Q(s, a: 0) (s)
(s5 0) Vasa + ya.
Vea+ + Vs.a Q" a)
TAE
TEE
OD
(4)
where Vs,aa = ED| r+, maxa Q(s', a'; 0) is the estimated target and Us.a Ep [r + 7 max, Q(s', a'; 0) represents the true target with A = arg ming E,, ((Q a; 0) Ve,a)* )
3.2. Extrapolation Error in MARL
Target Approximation Error (TAE) captures the discrep- ancy between the learned Q(ss, a; 0) and its target V/..a This error can be attributed to factors such as the inexact min- imization of 0 through gradient descent and the limited representational capacity of neural networks. Target Esti- mation Error (TEE) measures the difference between the estimated target ys.a and the true target Us.a. which can be influenced by issues such as overestimation and extrapo- lation errors. It is important to note that prior work often refers to this term as "overestimation error". However, over- estimation assumes that the error on the Qttaret follows uniform distribution. In this paper, we the term TEE to more accurately reflect that the error originates from the Qttarget itself, which is typically not uniformly distributed and may be dominated by extrapolation errors in MARL (see See. 3.2). Optimality Difference (OD) quantifies the gap between the value function of the current policy and that of the optimal policy. Unlike TAE and TEE, which depend on the current Q-function approximation with 0, OD pertains solely to how well the value function fits the Bellman optimality equation.
Extrapolation error arises in RL. when the value function inaccurately estimates the value of actions that are unseen or rare (?). If state-action pair (s, a) is absent from the dataset, the Q-function, Q(s, a; 8). produces an uncertain prediction due to the neural network's 's inability generalize accurately for unseen combinations. Specifically, in the Q- function update, while the transition (s, r,,)'') is sampled from the dataset, the next action a' generated by the Q- function or policy may be unseen or rare. This can lead to significant errors in estimating Q(s', propagating poor estimates to subsequent values.
Extrapolation error is most commonly associated with of- fline RL (Fujimoto et al., 2019; Levine et al., 2020), where the agent operates on a fixed dataset and cannot interact further with the environment, increasing the likelihood that the policy will sample actions not present in the dataset. In contrast, online RL allows the policy to interact with the environment and collect new data for the actions it gener- ates, enabling self-correction and reducing extrapolation error. However, in MARL, extrapolation error remains significant issue even in online settings. There are two main reasons for this. First, the exponentially growing joint ac- tion space makes the network increasingly susceptible to erroneous extrapolations, even in online RL (Fujimoto et al., 2023). Second, value factorization has a different property related to extrapolation error, which we will discuss in the next subsection.
Previous research in MARL has often focused on TAF, ex- ploring ways to improve target fitting. For example, in transitioning from a centralized Q-function to factorized Q-function, the function Q(s, a; 0) is simplified as a fune- tion of (s a;; 0))) which facilitates target approximation and reduces TAE. However, early value factorization meth- ods were found to be insufficiently expressive, meaning that they could not minimize TAE to zero. As a result, much of the research on value factorization has concentrated on improving the representational capacity of the Q-function. Nevertheless, recent empirical studies (Yu et al., 2022; Ellis et al., 2023; Hu et al., 2023) suggest that improving represen- tational capacity alone does not always lead to performance gains, particularly in complex domains.
For example, consider 5 agents with 10 possible actions each. The joint action space would then contain 105 possi- ble action combinations. To cover all possible joint actions for a specific state using uniform sampling, over 10 sam- ples would be required. In practice, this is rarely achievable with limited samples, meaning the Q-target Q(s', a') must often be estimated via extrapolation. It is noteworthy that extrapolation error is more severe for actions than for states. States tend to have more similarity in Euclidean space, mak- ing it easier for neural networks to extrapolate. In contrast,
The core issue lies in the tendency of prior research to simplify Q-function learning as a regression problem or to focus solely on theoretical properties in single-step matrix
3

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
discrete actions, often represented as one-hot vectors, lack such Euclidean similarity, which makes extrapolation diffi- cult. This issue is especially pronounced in MARL, where the joint action space grows exponentially, increasing the challenge of accurately estimating Q-values.
individual agents, rendering it ineffective addressing ex- trapolation error. To tackle this, we introduce the concept of Error Propagation Consistency (EPC):
Definition 3.1 (EPC), In value factorization methods, for a joint value function Q(s, a), if its corresponding individual utilities [Qi(s, as as)li=1 satisfy that, overestimation of ((, a) will result in the overestimation of all Qi (s,a.). we say that the factorization structure adheres to EPC.
Spread
5m vs 6m
Extrapolated Value
100%
Central-O
QNIX
80%
EPC is critical for mitigating TEE in value factorization methods. More importantly, if techniques are applied to reduce the error in the joint Q-function, EPC ensures that these improvements propagate to the individual utilities. This principle forms the basis for addressing TEE in the next section.
2.
0%
0.0
0.5
1.0
2M
Step
Step
In essence, factorization structures adhering to EPC must exhibit monotonicity, as stated in the following proposition:
Figure 2. (left) Proportion of extrapolated values for tasks with different numbers of agents. (right) TEE/TAE comparison for centralized Q-function and QMIX.
Proposition 3.2. Monotonicity, expressed as (Q.(s.a) 2 0, stands as a sufficient and necessary condition for EPC.
aQ(a,a)
To illustrate this issue, we present experimental results on the MPE and SMAC environments. Fig. 2(left) shows the proportion of extrapolated values in tasks with varying num- bers of agents. The proportion is calculated based on the fraction of (s', a') pairs in each update that are absent from the replay buffer. As shown, in the single-agent case, ex- trapolation is not a significant issue, as the proportion of extrapolated values quickly decreases to zero. However, as the number of agents increases, the exponentially growing joint action space leads to a substantial reliance on extrap- olated Q-targets. Fig. 2(right) compares TEE and TAE for a centralized Q-function and QMIX. As observed, TEE is larger than TAE and does not decrease significantly over time, as the large joint action space requires an impractically high number of samples to cover all possible actions. More- over, although factorized Q-functions like QMIX demon- strate improved scalability over centralized O-functions (as shown in Fig. 1). they do not resolve the TEE issue. This indicates that extrapolation error is a common challenge for all existing off- off-colicy MARL algorithms.
To illustrate, consider gradient descent on the objective function min [[ f(Q,, .... Qn))²), where represents the TD target and Q = f(Q...... Qx) denotes the factorized joint value function. The update rule of individual utility on (s, a) is given by:
Qi(s, + Qi(s,,))
0f 2aly - Q(s, (n(s, a, iQ, 8.00
where a is the learning rate. For a monotonic f with af 0, if the target y exceeds the current value function, i.e., y f(Q1..... Qn). the individual utility Q, will increase for (s. 0.). This means that overestimation of the target Q-fucction leads to overestimation of all individual utilities Qi(s, a.). Since actions are selected individually based on Qi(s, a.). this overestimation increases the probability of selecting a; for each i, thereby making the joint action a (a1...., an) under state more likely. This results in the corresponding trajectory with poor estimates being revisited more frequently, correcting the erroneous estimation.
> 8Q;
3.3. Error Propagation of TEE
Another challenge in online MARL is that the joint Q- function does not directly dictate the behavior of individual policies in value factorization methods, distinguishing it from single-agent RL.
Conversely, for a non-monotonie function, a larger target y may lead to smaller values for some Qi- Consequently, the likelihood of selecting the joint action a may decrease because underestimation of Qs o;) lowers the probabil- ity of selecting a;. In such cases, the erroneous estimation cannot be corrected, and consistently using this erroneous value as the target exacerbates error accumulation. Simi- lar observations are reported in Peng et al. (2021) and Hu et al. (2023), where non- mon-monotonic factorization, despite its better expressiveness, tends to underperform compared to monotonic counterparts, particularly in complex tasks.
As discussed earlier, online RL can mitigate extrapolation er- rors by increasing the frequency of executing unseen actions in the environment. However, in online MARI., individ- ual policies are derived from individual utilities rather than directly from the joint Q-function. Specifically, overestima- tion of the joint function does not necessarily increase the probability of selecting the corresponding actions for
4

---

Unlike prior works that impose additional assumptions on Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
4. Addressing TEE
A=0.0
A=02
A=0.4
A=0.8
zerg5
terranS vs
In this section, we introduce several techniques to mitigate TEE. As discussed previously, TEE arises primarily from extrapolation over the large joint action space in Q(s', a'). EPC provides a mechanism directly control the joint Q- function error in value factorization methods. Consequently, we propose the following approaches to address TEE: 1) Re- duce reliance on approximated Q(s', a') ) by employing its Monte Carlo estimation instead. 2) Mitigate uncertainty in rarely observed inputs by learning multiple Q(s', a') fune- tions and averaging them. 3) Restrict the representation of joint actions to reduce their impact. These techniques generally applicable across existing off- aff-policy MARL methods. In the following subsections, we elaborate on the methods inspired by these strategies.
Mean Test Win Rate
0.4
0.2
0.0
5M
SM
Step
Step
Figure 3. QMIX'S performance SMACv2 with different A.
This proposition establishes that TEE and its propagation depend on A. A larger reduces error, consistent with the intuition that À balances collected returns and learned value functions in target estimation.
4.1. Annealed Multi-Step Bootstrapping
However, a larger À is not always beneficial. With fixed behavior policy 14, the PQL operator converges to biased policy (Harutyunyan et al., 2016). While convergence to the optimal policy is possible as approaches и (Kozuno et al., 2021), this requires extensive timesteps and may not hold in practice. As shown in Fig. 3, while larger values improve initial training efficiency, their performance may degrade over time, ultimately lagging behind smaller À values.
To reduce reliance on Q(s',a') a') in TD targets, we propose replacing it with multi-step returns (Sutton & Barto, 2018). Multi-step targets derived from unbiased samples can off- set the effects of an undertrained value network, thereby reducing estimation error.
A prominent multi-step method for Q-learning is Peng's Q()) (PQL.) (Peng & Williams, 1994). The PQL. operator, N applicable to any policies μ and is defined as:
To capitalize on PQL's initial efficiency while avoiding bi- ased final performance, we propose À annealing strategy. This approach leverages PQL's error-reducing properties at the start of training with a large initial X, then gradually anneals to prevent policy bias. Specifically, we anneal À from 1 to X over the course of training, where X is a hyperparameter. The detailed annealing scheme is provided in Appendix D.1.
N Q -1 (1 A) Σ (1T
(5)
where A E [o, 1]. PQL is widely adopted in existing MARL algorithms (Peng et al., 2021; Zhang et al., 2021b; Hu al., 2023), yet its properties remain underexplored and merit further examination.
4.2. Averaged TD Target
Considering the error at iteration & is es, we consider the following update with error propagation:
To directly reduce TEE, common approach involves using multiple independent Q-function estimators and applying ensemble methods. Ensemble techniques are well-known for reducing variance and improving robustness (Ganaie et al., 2022). Consider M independently estimated Q-functions: Q a: 0i) (s, a) + (3 a), with et representing the error term, assumed be i.i.d across for each fixed state-action pair. By averaging these Q-functions, the variance of the target estimation is reduced proportionally to 1/M:
Thk E G(Qx) and Qk+1 == N (Qu + ex),
(6)
where is maintained as fixed behavior policy (discussed later). The following proposition illustrates PQL's error- reducing properties:
Proposition 4.1. The target estimation error for each up- date step k satisfies:
11Q+11 - ANH. S Be,
M Σ Q( a: 0²)00
where ||e and B 201-11 related to e is O ((1--111))))))1))
I-
The accumulated error
Var M
M Varle'
(8)
This reduction leverages the i.i.d. nature of the error e¹.
Proof. The proof follows from Kozuno et al. (2021); full details are provided in Appendix B.
5

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
error properties (Anschel et al., 2017: Chen et al., 2021), we do not make such assumptions. Instead, we acknowledge that the limited expressiveness of value factorization meth- ods can introduce model bias into e², stemming from TAE, as discussed in Section 3. Additionally, due to these biases, we avoid using conservative methods like those employed in offline RL (An et al., 2021; Bai et al., 2022). Further details are provided in Appendix D.2.
For methods that directly use the joint Q-function, such as MADDPG, we apply function g to map the joint action space to smaller, more manageable space. The g(a) is modeled as a neural network that takes the joint action a as input and outputs a vector of categorical variables. This net- work is optimized using seraiggttthrouge gradients (Bengio et al., 2013). For example, in MPE with 4 agents, the joint action space has a size of 5 By using the mapping function g(a), we reduce this space to 2 through 10-dimension vector. This vector is comprised of 5 categorical distribu- tions, each with 2 possible classes. We then sample from g a) and convert the result into a one-hot vector. The target Q-function is then given by: onehoot onehot(draw ((ra((((a))) This ensures that, instead of having 5 Q-values at each state s', we only have 2 values, thus reducing the extrapolation in the target Q-function.
In this approach, we propose directly averaging Q-functions within the TD target, incorporating the action space. Specif- ically: for methods that only utilize joint f-fuccioos,, we utilize M joint Q-function: Q(s, a: 0) = > Q(ss, a; 0)); for value factorization methods, we utilize M individual util- ity: Q(s, az 0, v) Σラ= Q(, [Q:(s, as; D])] v where d represents the parameter of the mixing network. The corresponding loss function can be written as:
The corresponding results are shown in Fig. 4(left), where the "MADDPG-abl" curve represents the ablation of using highlighting that simply mapping the joint ac- tion to g(a) does not lead to performance improvement. The key reason is that mapping the action space to categorical g(a) restricts the representation of the joint action while continuous g(a) cannot.
M
L(O,0) = Σ ED [(,
,)) - yo )),
where Vs is the PQL target using averaged Q-function. For value factorization methods like QMIX, it is worth noting that there is no need to average multiple mixing networks, as they do not depend on the action space. Further details are provided in Appendix D.2. Another critical point is that the theoretical guarantees underlying these results implicitly assume that the factorization satisfies the EPC condition, i.e., monotonicity. This requirement is essential because both PQL and ensemble methods aim to control the error of the joint Q-function, and only a monotonic factorization can propagate this control to individual utilities effectively.
For methods like QPLEX, which utilize joint actions as an individual component, we apply simpler technique to restrict the influence of joint actions. QPLEX relies on a weight parameter (s a). which can become erroncous due to the vast size of the joint action space. As shown in Fig. 4(right), the dashed line and the right y-axis display the mean, std and maximum values of QPLEX's ' X during train- ing. We observe that while the mean and std of X; remain small and stable, the maximum value grows significantly over time. This indicates the presence of outliers in X; likely due to error accumulation during training. This error accu- mulation leads to instability around the 5M training step, causing a significant drop in QPLEX's performance.
4.3. Restricted Action Representation
Since different methods utilize the joint action space in varying ways, we illustrate the application of the Restricted Action Representation (RAR) technique using MADDPG and QPLEX as examples. While the specific methods differ, they share the common objective of reducing the influence of joint actions in the TD target.
The reason for this issue is straightforward: as A grows large at rarely encountered state-action pairs (s, a), erro- neous values accumulate over time. This eттor persists be- cause the probability of selecting action a does not increase with (s a). This is similar to the issue discussed in Sec- tion 3.3. To mitigate this, we propose directly restricting A; in QPLEX. Specifically, we bound by applying the Sigmoid function: X₃ a) = Sigmoid(A{(s, a)), which ensures that A E [0, 1]. The results in Fig. 4(right) show that QPLEX-RAR significantly improves the stability of QPLEX, demonstrating that RAR effectively reduces the negative impact of the joint action space.
spread
zerp5 vs
Mean Test Win Rate
0.5
2_mean
A std
12
Reward
-220-
0.4-
-340
MADOPG MADOPG-RAR
0.2-
MADDPG-AM abl
0.0
0.0
1.0
10M
Step
Step
Figure 4. The effect of RAR MADDPG (left) and QPLEX (right).
6

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
AEFACMAC
FACMAC
AEMADOPG-NAR
MADDPG
Mean Test Win Rate
V5_ 642g
MMM2
27m_ys 30m
1.0
0.8-
0.6
2M 0
2M
2M 0
2M
Step
Figure Mean test win rate of AEFACMAC, AEMADDPG RAR, FACMAC and MADDPG on SMAC.
AEQMIX
QMIX
zerg5 5
zerg10 10
zerg20 vs
zerg10 vs
zerg20 23
0.8
0.6
0.2
Mean Test Win Rate
protoss5 vs
protoss10 vs_
protoss20 20
protoss10 vs 11
protoss20 vs 23
0.8-
0.6
0.4-
0.2
terran5_vs vs_5
terran10 vs_10
terran20 vs 20
terran10 vs
terran20 v5_23
0.6
0.2
0.0
10M 0
10M
10M
10M 0
10M
Step
Figure Mean test win rate of AEQMIX and QMIX on SMACv2.
5. Experiments
and two Super Hard maps. The codebases for FACMAC and MADDPG are adopted from Peng et al. (2021). The results are shown in Fig. 5, where we observe that AEFACMAC and AEMADDPG-RAR significantly improve performance over their base algorithms. The performance gain is par- ticularly notable in the two Super Hard maps with larger joint action spaces, indicating that our methods effectively address the challenges posed by the increased joint action space. Despite MADDPG's generally poor performance in SMAC, our approach allows it to perform well on the Easy and Hard maps and even enables learning on the Super Hard maps, where MADDPG typically struggles.
In this section, we evaluate the performance of the proposed approach across several popular domains, including SMAC (Samvelyan et al., 2019) and SMACv2 (Ellis et al., 2023). Due space limitations, we primarily focus on QMIX (Rashid et al., 2020b) in SMACv2, and FACMAC (Peng et al., 2021) and MADDPG (Lowe et al., 2017) in SMAC, as they represent value-based and policy-based off off-policy MARL methods, respectively. We refer to the algorithms with annealed multi-step bootstrapping and averaged TD- target as Annealed Ensemble QMIX (AEQMIX), AEFAC- MAC, and AEMADDPG- RAR. Further results, including AEQMIX and QMIX SMAC and GRF (Kurach al., 2020), AEVDN, EEQPLEX-RAR, VDN (Sunehag al., 2018), and QPLEX-RAR on SMACv2, as well as details regarding hyperparameters and experimental settings, can be found in Appendix D.
SMACv2. SMACv2 was introduced to overcome some of the limitations of SMAC, such as the lack of stochas- ticity and partial observability (Ellis et al., 2023). Unlike SMAC, SMACv2 incorporates randomly generated units and initial positions, which introduce more stochasticity and create challenging scenarios. We evaluate QMIX on SMACv2, as it is the SOTA method for this domain but still requires improvement, especially for tasks with large agent numbers. The codebase is from Pymar12 (Hu et al., 2023). As shown in Fig. 6, we test the algorithms on 15
5.1. Main Results
SMAC. We evaluate on four maps from the FACMAC paper (Peng et al., 2021), including one Easy map, one Hard map,

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
QMIX-64
QMIX-128
QMIX-256
zerg5
Mean Test Win Rate
zerg5 $__________ vs_5
protoss5 vs
terran5 _s_ $_5
Mcan Test Win Rate
0.6
0.8
0.4-
0.6
QMIX
0.4
0.2
2QMIXIM-2)
40MIXIM=4)
EQMIXIM-8)
0.2
0.0
8E
0.0
10M
10M 0
10M
10M
Step
Step
Figure 7. Ablations of QMIX with different X, M and hidden_size
QPLEX
QPLEX-RAR
AEMADDPGRAR
AEMADDPG
Mcan Test Win Rate
protosss v5_5
terran5 vs_5
Mean Test Win Rate
2c_vs.644
MMM2
27m. v5..vs.l 30m
1.0
0.6
0.8
0.4
0.2-
02
10M 0 Step
10M
2M
2M
2M
Step
Figure 8. The effect of RAR QPLEX and MADDPG.
maps of SMACv2. QMIX struggles to achieve high win rates across several tasks, particularly in the 10_vs_/l and 20 23 scenarios. However, AEQMIX exhibits signifi- cant improvements across all maps, especially on the more challenging tasks where QMIX's win rate is notably low. These results demonstrate the effectiveness of our proposed method in improving existing SOTA algorithms and tackling challenging environments.
with the well-established property of ensemble methods to decrease variance by a factor of 1/M.
Figure 7 (left) illustrates the effects of ensemble size and annealing performance, where AQMIX represents QMIX with annealing, and EQMIX represents AEQMIX without annealing but with ensemble size M. First, we observe that larger ensemble sizes consistently improve performance, demonstrating the benefits of variance reduction through av- eraged largets. Second, comparing AEQMIX with EQMIX under the same M shows that annealing generally enhances performance. Finally, the combined approach of annealing and averaging results in a mutually beneficial effect, leading to significant performance improvements. It is worth noting that simply increasing the number of parameters does not guarantee better performance. For example, Figure 7 (right) shows the performance of QMIX with varying hidden sizee of its networks. Here, performance worsens the network size increases, suggesting that larger networks more prone to extrapolation errors. This implies that performance gains arise not from an increase in model parameters, but from more accurate target estimation.
5.2. Ablation Studies and Discussions
In this subsection, we conduct further experiments to ana- lyze the impact of the techniques introduced our paper.
Target Estimation Error
2.4 145
larget Network Variance
2.0
1+:2
Anneeai
A-0.0
1.9-
An02
A-0.4
1.5
H=5
A-0.6
1.4
1.0
0.9
0.5
0.4
0.0
Step
Step
Figure shows the effect of RAR on QPLEX and MADDPG. By restricting the action representation, the performance of QPLEX stabilizes, and MADDPG's performance improves. While this technique limits the expressive power of the action space, it highlights that full expressiveness is not always necessary. especially when the joint action space large. In such cases, overly expressive models may suffer from estimation errors, negatively impacting performance.
Figure 9. TEE and its variance with different À and M.
Figure shows the impact of varying À and ensemble size M on TEE. As expected, larger X results in lower TEE. Our À annealing approach keeps TEE low throughout train- ing, promoting efficient learning in the early stages while mitigating bias in the later stages. Additionally, increas- ing the ensemble size M reduces the variance, consistent
8

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
6. Conclusion
Chen, X., Wang, C., Zhou, Z., and Ross, K. W. Randomized ensembled double q-learning: Learning fast without model. In 9th International Conference
This paper highlights the oveen-overlooked issue of TEE in off-policy MARL. We identify that TEE primarily arises from extrapolation errors in the large joint action space of the joint Q-function and provide detailed analysis of its propagation and reduction. We propose three techniques to mitigate TEE from different perspectives, which are broadly applicable to a wide range of off-policy MARL methods. Our experimental results demonstrate that the proposed methods can significantly improve the performance of both policy-based and value-based off-policy MARL algorithms, including factorized and factorier approaches. We believe that our findings open up new avenues for future research and offer fresh perspective on addressing the challenges of TEE in off-policy MARL.
Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Ellis, B., Cook, J., Moalla, S., Samvelyan, M.. Sun, M.. Mahajan, A., Foerster, J. N., and Whiteson, S. Smacv2: An improved benchmark for cooperative multi-agent re- inforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Foerster. J., Farquhar, G., Afouras, T.. Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradi- ents. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Acknowledgements
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep re- inforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learn- ing, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 2019.
This work was supported by NSFC under Grant 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
Impact Statement
There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fujimoto, S., Chang, W., Smith, E. J., Gu, S., Precup, D., and Meger, D. For SALE: state-action representation learning for deep reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
References
Agarwal, R., Schuurmans, D., and Norouzi, M. An opti- mistic perspective offline reinforcement learning. In International Conference on Machine Learning, pp. 104- 114. PMLR, 2020.
Ganaie, M. A., Hu, M., Malik, A., Tanveer, M., and Sugan- than, P. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
An, G., Moon, S., Kim, J.H and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018
Anschel, O., Baram, N., and Shimkin, N. Averaged-dqn: Variance reduction and stabilization for deep reinforce- ment learning. In International conference on machine learning, pp. 176-185. PMLR, 2017.
Harutyunyan. A., Bellemare, M. G., Stepleton, T., and Munos, R. Q with off-policy corrections. In Inter- national Conference on Algorithmic Learning Theory, pp. 305-320. Springer, 2016.
Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty-drive offline reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Hu, J., Wang, S., Jiang, S., and Wang, M. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. In The Second Blogpost Track at ICLR 2023, 2023.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint 10X.v:130..3432. 2013.
Hüttenrauch, M., Šošić, A., and Neumann, G. Deep rein- forcement learning for swarm systems. J. Mach. Learn. Res.,
January 2019. ISSN 1532-4435.

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
Kozuno, T., Tang, Y., Rowland, M., Munos, R., Kaptur- owski, S., Dabney, W., Valko, M., and Abel, D. Revis- iting peng's q X for modern reinforcement learning. In International Conference on Machine Learning, pp. 5794-5804. PMLR, 2021.
Rashid, T., Farquhar, G., Peng, B., and Whiteson, S. Weighted qmix: Expanding monotonie value function factorisation for deep multi-agent reinforcement learning, Advances in neural information processing systems, 33: 10199-10210, 2020a.
Kraemer, I.. and Banerjee, B. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neuro- computing, 190:82 94, 2016.
Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, (1()::2334 7284, 2020b.
Kurach, K., Raichuk, A., Stañezyk, P., Zajac, M., Bachem, O., Espeholt, L.,, Riquelme, C., Vincent, D., Michalski, M.. Bousquet, O., et al. Google research football: A novel reinforcement learning environment. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4501-4510, 2020.
Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C., Torr, P. H. S., Foerster, J. N., and Whiteson, S. The starcraft multi- agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Sys- tems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, pp. 2186-2188. International Foundation for Au- tonomous Agents and Multiagent Systems, 2019.
Lee, K., Laskin, M., Srinivas, A., and Abbeel, P. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pp. 6131-6141. PMLR, 2021.
Schäfer, L., Slumbers, O., McAleer, S., Du, Y., Albrecht, S. V., and Mguni, Ensemble value functions for effi- cient exploration in multi-agent reinforcement learning. arXiv preprint arXiv:2302.03449 2023.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2rXiv:2005.01643 2020.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, Multi-agent actor-critic for mixed cooperative-competivv environments. Advances in neu- ral information processing systems, 30, 2017.
Singh, A. J., Kumar, A., and Lau, H. C. Hierarchical multiagent reinforcement learning for maritime traffic management. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Sys- tems, AAMAS '20, Auckland, New Zealand, May 9-13, 2020, pp- 1128-1 International Foundation for Au- tonomous Agents and Multiagent Systems, 2020. doi: 10.5555/3398866133399999
Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap,
Son, K., Kim, D.., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Inter- national conference on machine learning, pp. 5887-5896. PMLR, 2019.
T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928- 1937. PMLR, 2016.
Oliehoek, A. and Amato, C. concise introduction to decentralized POMDPs. Springer, 2016.
Son, K. Ahn, S., Reyes, R. D., Shin, J., and Yi, Y. Qtran++: improved value transformation for coopera- tive multi-agent reinforcement learning, arXiv preprint arXiv:2006. 12010, 2020.
Oliehoek, F. A., Spaan, M. T., and Vlassis, N. Optimal and approximate q v-luu functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
Su, J., Adams, S., and Beling, P. Value-decomposition multi- agent actor-critics. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1 11352-11360, 2021.
Peng, B., Rashid, T., Schroeder de Witt, C., Kamienny, P.-A., Torr, P., Böhmer, W., and Whiteson, S. Facmac: Factored multi-agent centralised policy gradients. Ad- vances in Neural Information Processing Systems, 34: 12208-12221, 2021.
Sunehag, P., Lever, G., Gruslys, A., Czamecki, W. M., Zambaldi, V. F,, Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., and Graepel, T. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and
Peng, J. and Williams, R. J. Incremental multi-step q- learning. In Machine Learning Proceedings 1994, pp. 226-232. Elsevier, 1994.
10

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
MulliAgen Systems, AAMAS 2018, Stockholm, Sweden, July 10- 15, 2018, pp. 2085-2087. International Founda- tion for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018.
Nieves, N. P., Ni, Y., Banijamali, S., Cowen-Rivers, A. I., Tian, Z-., Palenicek, D., Bou-Ammar, H., Zhang, H., Liu, W., Hao, J., and Wang, J. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. In Conference on Robot Learning, 2020.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Wang, J. Ren, Z., Han, B., Ye, J., and Zhang. C. Towards understanding cooperative multi-agent q-learning with value factorization. Advances in Neural Information Pro- cessing Systems, 34:291 012229142-22 142-29155 2021a. la.
Wang, J., Ren, Z., Liu, I., Yu, Y., and Zhang, C. QPLEX: duplex dueling multi-agent q-learning. In 9th Interna tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021b.
Wang. Y., Han, B., Wang, T., Dong, H., and Zhang, C. DOP: off- off-policy multi-agent decomposed policy gradients. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net, 2021c.
Xu, Z., Li, D., Bai, Y., and Fan, G. Mmd-mix: Value function factorisation with maximum mean discrepancy for cooperative multi-agent reinforcement learning. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2021.
Yang, Y. Hao, J., Liao, B., Shao, K., Chen, G., Liu, W., and Tang. H. Qatten: A general framework for cooper- ative multiagent reinforcement learning. arXiv preprint 20Xiv:2002.03939, 2020.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, The surprising effectiveness of ppo in cooper- ative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24622 2022.
Zhang, T., Li, Y., Li, S., Ye, Q.. Wang, C., and Xie, G. De- centralized circle formation control for fish-like robots in the real-world via reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 8814-8820. IEEE, 202
Zhang, T., Li, Y., Wang, C., Xie, G., and Lu, Z. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Con- ference on Machine Learning, pp. 12491-12500. PMLR, b021b.
Zhou, M., Luo, J., Villela, J., Yang, Y., Rusu, D., Miao, J., Zhang, W., Alban, M., Fadakar, I., Chen, Z., Huang, A. C., Wen, Y., Hassanzadeh, K., Graves, D., Chen, D., Zhu, Z., Nguyen, N. M., Elsayed, M., Shao, K., Ahilan, S., Zhang, B., Wu, J., Fu, Z.., Rezaee, K., Yadmellat, P., Rohani, M.,

---

Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning
A. Related Works
Value factorization. In addition to the previously discussed methods, various approaches address the challenge of value factorization. For value-based methods, Qatten (Yang et al., 2020) employs an attention mechanism augment the expressive capacity of linear factorization. WQMIX (Rashid et al., 2020a) improves QMIX's expressive ability through the incorporation of weighted operator and true value network. QTRAN+ (Son et al., 2020) refines the constraints of QTRAN to improve efficiency. For policy-based methods, VMIX (Su et al., 2021) applies QMIX's factorization to the value function of A2C (Mnih et al., 2016). DOP (Wang et al., 2021c) utilizes linear factorization on the Q-function of COMA (Foerster et al., 2018), while FOP (Zhang et al., 2021b) extend QPLEX's factorization to soft actor-critic (Haarnoja et al., 2018) framework. FACMAC (Peng et al.. 2021) combines QMIX's 's factorization with MADDPG (Lowe et al., 2017). These approaches primarily concentrate on enhancing factorization itself, specifically addressing the TAE problem introduced in this paper, without delving into the underlying reasons for the success of value factorization. While theoretical papers such as (Wang al., 202 take steps to unveil the efficiency and credit assignment of value factorization, they lack substantial support for subsequent improvements. Recent examinations of these methods (Yu et al., 2022; Ellis et al., 2023; Hu et al., 2023), along with more comprehensive experiments, highlight QMIX as the most popular and robust value-based MARL algorithm. Therefore, distinct from previous approaches, our work approaches value factorization from a novel perspective, introducing further enhancements to existing methods.
Ensemble RL and MARL. Our analysis is similar to Averaged-DQN (Anschel et al., 2017), which ensembles the Q- functions from the past M steps. Despite proving effective in variance reduction, Averaged-DQN relies on assumptions that may not always hold in practice. Other ensemble methods (Lee et al., 2021) incorporating the standard deviation of Q-functions were not discussed here due to limited observed improvements and their divergence from the main focus of this paper. REDQ (Chen et al., 2021) employs in-target minimization across a random subset of Q-functions from the ensemble. However, this approach proves unsuitable for value factorization, possibly due to the presence of model bias (See Appendix D.2). In MARL, EMAX (Schäfer et al., 2023) applied a similar ensemble method on VDN and QMIX with UCB and majority vote to improve exploration. MMD-MIX (Xu al., 2021) introduce REM (Agarwal et al., 2020) into a distributional view of QMIX be more robust in randomness. These methods do not explicitly consider the extrapolation error.
Offline RL. The in-target average ensemble employed in our paper bears resemblance to the approach used in offline RL (Agarwal et al., 2020; Fujimoto et al., 2019; Levine et al., 2020). Additionally, the in offline RI. aligns with the extrapolation error addressed in this paper. As a result, similar methods may yield comparable effects. However, different from An et al. (2021); Bai et al. (2022), our paper does not require a more conservatieeppessiiiste estimation of the target, as the monotonic constraint enables self-correction in online RL. Moreover, we found that any degree of pessimism negatively impacts performance, as detailed in the Appendix D.2.
(OOD) action studied
B. Proof of Proposition 4.1
Lemma B.1 (HHarutyunyan et al., 2016)). The PQL operator can be rewritten in the following
Ns Q = -I- (I- +
(1- Q)
(10)
Using this lemma, we have:
N (Qx + +x)== = (I (I---hh 2 7 (Q) + ex)l
=N Qx + 711 - A)(I 7XPH)-1pp ek
As result,
11Q+1 - NN
Qulloo = 7(11- - -I -XPPP) -1r
< 7(1 - A)
1- ||e1|100 = Be.
This shows the propagation of TEE relative to on each step.
For the algorithm:
Tk G(Qu) and Qx+1 NRt. Qx Ek
12

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
Algorithm 1 AEQMIX
1: Initialize M action- value networks for all agents {[Qi(Ti, 44; (1)) 1i=1 with parameter O and a mixing hypernetorr
H with parameter v
2: Initialize target networks: v/ = 15, = 0
3: Initialize replay buffer D = {
4: while k - episode_mar do
5: set trajectory buffer T []
6: for = to max epsode length do
7: Explore using E greedy with (T) Σ م Qi(T.:;0)))
8: Store transition (st, Tı, az Tt, S1+1, T8+1) into T
9: end for
10: Store trajectory into D and sample a mini-batch b
for each trajectory T in b do
for each transition (s, T, a, T') in T do
13:
Form joint action a' by d; arg max (i(
14:
Compute target joint value Q'(s', a') using (??) end for
15:
16:
Compute PQL. target Vs,a with À using Q'(s', a')
17:
Compute joint value Q(s, a; 00, vb)
end for
Compute loss through (9)
20:
Adam updates 0, 1b with the computed loss if k mod d 0 then
21:
22:
Update target networks: v// 1/, 0' Update À through (13)
23:
24:
end if
4: l
end while
(Kozuno et al., 2021) introduced the following lemma: Lemma B.2 (Kozuno et al., 2021)), For any K the following holds:
K g
|VP - VPx < O(3*)
2
Rk-k-1
k- 11x1l0o
(11)
1-
k=0
where PK Ap + (1 A)TT, Pt dp + (1 A)TT and n E G(QPt).
Therefore, in this paper, we have
K-1
V - |Px| < O(3 + 1-7 km0
Σ
3k 3k-k-1 Be
(12)
The second term represents the eтror dependence which can be futher written as:
2
K 1
9K k
23 1 3K
c 0 71 21) 2))
000002220
(1 (1-7)2 e).
1 7 1 3
k=0
This completes the proof.
C. Pseudo code
The pseudo code of AEQMIX is summarized in Algorithm
13

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
D. Experimental Details
D.I. Hyperparameters
Our implementation of VDN, QMIX and QPLEX is based on the pymar12 (Hu et al., 2023) code base. All hhyerrparameter used in our algorithm is consistent with QMIX except for the additional X and ensemble size M, as presented in Table 1. For stability, d is updated alongside the fixed target network follows:
A-
A x + 1+ ak.
(13)
where a = 10/TT, and T is the total number of environmental steps for training. While this scheme is heuristic, the algorithm is not sensitive to the specific annealing schedule, as long as À converges to X". The implementation of MADDPG and FACMAC directly taken from Peng et al. (2021), without any parameter adjustment.
Table 1. Hyperparameters used for SMAC, SMACv2 and GRF
hyperparameters
SMAC
SMACv2
GRF
Action Selector
epsilon greedy
epsilon greedy
epsilon greedy
start
1.0
1.0
1.0
6. finish
0.05
0.05
0.05
Anneal Time
100000
100000
100000
Runner
parallel
parallel
parallel
Batch Size Run
4
32
Buffer Size
5000
5000
2000
Batch Size
128
128
128
Optimizer
Adam
Adam
Adam
Target Update Interval
200
200
200
Mixing Embed Dimension
32
32
32
Hypernet Embed Dimension
64
64
64
Learning Rate
0.001
0.001
0.0005
0.6
0.4
0.8
{0.0, 0.4)
{0.0, 0.2)
0.8
Ensemble Size
8
8
2
Gamma
0.99
0.99
0.999
RNN Hidden Dim
64
64
256
D.2. Additional Discussion on Averaged Target
In this section, we provide further discussion on the averaged Q-target.
First, we examine the effect of using multiple mixing networks to compute the ensembled target:
a(s.a;0,0) = Σ Hs (i(s, Qn,s, ani Bn); )d).
(14)
As shown in Fig. 10(a), using more than one mixing network (M > 1) does not improve performance. This is because the error predominantly arises from the action space, while the mixing network only takes the state as input, which does not effectively address the action-related errors.
Next, we explore whether using more conservative TD target could improve performance, similar to the approaches used to address extrapolation error in offline RL (An et al., 2021; Bai et al., 2022). To this end, we implement target similar to REDQ (Chen et al., 2021):
y = r + minn minnQ(s, a; ())
(15)
JEM
where M is a set of M distinct indices from the ensemble 1,2,..., 10. The results, shown in Fig. 10(b), represent the win rates averaged across zerg 5 vs 5, protoss vs 5 and terran vs on SMACv2 at 3M time steps. As the level of
14

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
zerg_ VS __vs_ vs_
Mean Test Win Rate
0.6
with double q-learning
M=
0.584
without double q-learning
M =
Mean Test Win Rate
0.6
M=8
0,516
0.492
0.5
0.387
0.2
0.0
0
10M
0.2
M=0
M=0
M=2
M=3
M=4
M=5
Step
(a)
(b)
Figure 10. (a) Performance QMIX with different number of mixing network. (b) Comparison of the performance with different pessimism of the TD target.
pessimism increases along the x-axis, we observe that using a more pessimistic TD target does not improve performance and, in fact, has a negative impact. This in contrast to offline RL settings, where pessimism has proven useful for mitigating extrapolation errors. However, in online RL setting, this approach introduces unnecessary conservatism, which hinders learning and leads to suboptimal performance.
D.3. Additional Results SMAC, SMACv2 and GRF
AEQMIX
QMIX
1c3s52
2s3z
2s vs 1sc
8m
MMM
1.0
Mean Test Win Rate
0.0
2M
0
2M
0
2M 0
2M 0
2M
2c vs. 642g
3s5z
35, 5z
8m 9m
10m vs 11m
1.04
0.8
0.2
5M 0
5M
SM
0
5M
0
5M
3s52. vs. 3s62
v5.6m
vs. 82
corridor
MMM2
0.6
0.0
10M
10M
10M
10M
10M
Step
Figure 11. Mcan Test Win Rate of AEQMIX and QMIX on SMAC.
Fig. 11 shows the performance of AEQMIX and QMIX on SMAC. Fig. 12 shows the performance of AEQPLEX-RAN QPLEX-RAR, AEVDN and VDN on SMACx2. Fig. 13 shows the performance of AEQMIX and QMIX on GRE
15

---

Revisiting Cooperative Oft-Policy Multi-Agent Reinforcement Learning
REQPLEX-RAR
OPLEX-RAR
AEVDN
VDN
zerg5 ___
zerg10_ vs 10
zerg20_vs vs 20
zerg10 vs
0.8
0.6
0.4
0.2
Mean Test Win Rate
0.0
protoss5 vs 5
protoss10 vs 10
protoss20 vs 20
protoss10 vs 11
0.8
0.6
0.4
0.2
0.0
terran5 vs 5
terran10 vs
terran20 vs 20
terran10 vs
0.6
0.2
0.0
10M 0
10M Step
10M 0
10M
Figure 12. Mean Test Win Rate of AEQPLEX-RAN RARR OPLEX-RAR, AEVDN and VDN SMACv2.
AEQMIX
QMIX
Mean Test Score
vs. with keeper
counterattack easy
counterattack hard run_pass shoot with keeper
0.8
0.2
20M 0
20M 0
20M
20M 0
20M
Step
Figure 13. Mean Test Score of AEQMIX and QMIX GRF.
16


Reviews:
[{"review_id": "KV9nHvlwdr", "reviewer": "Reviewer_YK2e", "summary": "This paper identifies a problem of erroneous Q-target estimation, primarily caused by extrapolation errors, which worsens as the number of agents increases. The authors follow the previous work on single-agent error decomposition and apply it to multi-agent Q-learning, decomposing the error into Target Approximation Error (TAE), Target Estimation Error (TEE), and Optimality Difference (OD). To address the issue of TEE, the authors propose a suite of techniques, including annealed multi-step bootstrapping, averaged Q-targets, and restricted action representation. Experiments on SMAC, SMACv2, and Google Research Football show significant improvements over baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback, which has greatly helped us improve the quality of our paper. We were glad to hear that you found our claims convincing and that our proposed methods make sense for addressing the problem. Below, we address the key concerns you raised.\n\n> 1. The authors frequently mention the MPE environment in the introduction but do not include any experiments on it. Why was this environment omitted?\n\nThe MPE environment is included in **Figure 2 (left) and Figure 4 (left)**. However, we did not highlight it in the experiment section because it contains only one suitable task (Spread), and is significantly less complex than SMAC and SMACv2. We will provide additional results for Spread in the appendix in the next version. For futher details, please refer to point 5 of our rebuttal to reviewer o7jb.\n\n> 2. The paper should compare its approach with other research addressing overestimation in MARL, such as [1].\n\nThank you for the suggestion. We will add a comparison to RES.\nHowever, based on the current results, it appears that RES does not outperform the fine-tuned QMIX when evaluated within the PyMARL2 codebase. \nThis also aligns with our argument that extrapolation error does not always lead to overestimation. As shown in **Figure 10(a) (Appendix D.2)**, applying a more underestimated target actually degrades performance.\n\n> 3. The scope of paper is framed around off-policy learning, but the key techniques seem to target algorithms using Bellman optimality equations.\n\nOur techniques target methods that utilize Q-functions, which are fundamental to most off-policy methods. Since these methods require off-policy Q-target estimation, they inherently face target estimation error, as discussed in section 3 of our paper.\n\n> 4. The section on Restricted Action Representation in MADDPG is difficult to follow. The authors could provide a more intuitive example, such as one where the number of actions differs from the number of agents, rather than using the 5^5 example, which is not clearly explained.\n\nThanks for this suggestion, we choose the 5^5 to 2^5 example because it directly corresponds to our experimental setup in Figure 4 (left).\nFor another example such as 4 agents with 5 action, we can also compress the action space from 5^4 to 2^4.\nFor a more detailed explanation of RAR, please refer to point 1 of our rebuttal to reviewer Lbue.\n\n> 5. The paper references the $\\lambda$ parameter in QPLEX but does not provide a clear introduction to QPLEX itself. Some background should be added.\n\nThank you for pointing this out. We will add a brief introduction to QPLEX to provide the necessary background for the discussion on the $\\lambda$ parameter.\n\n> 6. The discussion on $\\lambda$ in QPLEX does not seem to align with the idea of Restricted Action Representation. Instead, it appears to be merely a constraint on the weight values rather than a true action space compression.\n\nThe concept of RAR is to **limit the effect of joint actions on the joint Q-function**.  In QPLEX, since the joint action component is already separated by $\\lambda$, directly constraining $\\lambda$ can effectively restrict its influence. Applying the same technique as in MADDPG could achieve similar results but would require additional networks. We will provide corresponding experiments in the appendix. \n\n> 7. The authors propose applying an additional sigmoid function to $\\lambda_i$, but the original QPLEX paper already applies sigmoid before summation. While this paper applies it after summation, the practical impact of this change is unclear.\n\nThe original QPLEX paper applies a sigmoid before summation, but this does not restrict the impact of joint actions, as other weight terms can still amplify it. The full expressive power of QPLEX requires $\\lambda\\in[0,\\infty]$, meaning that even with a pre-summation sigmoid, QPLEX retains full representation capability. By constraining $\\lambda\\in[0,1]$ , our approach explicitly limits action representation ability, leading to more stable results.\n\n> 8. The ablation study on Restricted Action Representation is confusing. The results for MADDPG and QPLEX should be presented together in the same figure for easier comparison. Why are the ablation studies conducted in different environments for different methods? This makes it difficult to interpret the results consistently.\n\nWe understand the concern. However, our MADDPG and QPLEX use **different codebases** as detailed in  Appendix D.1, making direct performance comparison within the same figure difficult (basically because the difference in **parallelized environments**). \nMoreover, the performance of MADDPG and QPLEX are significantly different, SMACv2 may be too hard for MADDPG and SMAC may be too easy for QPLEX, which makes the effect of RAR less noticeable.\nWe will clarify this in the next version.", "claims_and_evidence": "The claims are generally convincing.", "methods_and_evaluation_criteria": "- The proposed methods make sense for addressing the problem of erroneous Q-target estimation.\n- The benchmarks, including SMAC, SMACv2, and Google Research Football, are appropriate for assessing the performance of MARL algorithms.", "theoretical_claims": "The paper does not introduce particularly complex new theoretical contributions but rather focuses on experimental observations.", "experimental_designs_or_analyses": "- The authors frequently mention the MPE environment in the introduction but do not include any experiments on it. Why was this environment omitted?\n- The paper should compare its approach with other research addressing overestimation in MARL, such as [1].\n\n---\n\n[1] Regularized Softmax Deep Multi-Agent Q-Learning.", "relation_to_broader_scientific_literature": "The authors highlight the increasing error in joint-action estimation as the action space expands, which is a well-known issue in multi-agent reinforcement learning. They analyze this issue in detail and propose solutions, making it a relevant and important contribution.", "essential_references_not_discussed": "[1] Regularized Softmax Deep Multi-Agent Q-Learning.", "other_strengths_and_weaknesses": "Strengths:\n- The paper presents several effective improvements over baseline methods.\n- The experimental results demonstrate clear performance gains.\n\n\nWeaknesses:\n- The scope of paper is framed around off-policy learning, but the key techniques seem to target algorithms using Bellman optimality equations.\n- The section on Restricted Action Representation in MADDPG is difficult to follow. The authors could provide a more intuitive example, such as one where the number of actions differs from the number of agents, rather than using the 5^5 example, which is not clearly explained.\n- The paper references the $\\lambda$ parameter in QPLEX but does not provide a clear introduction to QPLEX itself. Some background should be added.\n- The discussion on  $\\lambda$ in QPLEX does not seem to align with the idea of Restricted Action Representation. Instead, it appears to be merely a constraint on the weight values rather than a true action space compression.\n- The authors propose applying an additional sigmoid function to $\\lambda_i$, but the original QPLEX paper already applies sigmoid before summation. While this paper applies it after summation, the practical impact of this change is unclear.\n- The ablation study on Restricted Action Representation is confusing. The results for MADDPG and QPLEX should be presented together in the same figure for easier comparison. Why are the ablation studies conducted in different environments for different methods? This makes it difficult to interpret the results consistently.", "other_comments_or_suggestions": "See Weaknesses", "questions_for_authors": "See Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FjOm6z588S", "reviewer": "Reviewer_o7jb", "summary": "The work studies the problem of overestimation and target estimation errors in off-policy multi-agent reinforcement learning (MARL). The work first outlines how action-value estimation in MARL often suffers from estimation errors as a consequence of the exponential growth of the joint action space. To substantiate the problem, the work decomposes the value error into three components and discusses how in particular target estimation error (TEE) can cause worse performance in MARL, and how error propagates in value decomposition algorithms. The work proposes three approaches to mitigate such target estimation error based on (1) multi-step target estimation, (2) computing averaged target values across an ensemble of value functions, and (3) projecting the large action space into lower dimensional discrete space to simplify learning action-value functions. The efficacy of these components applied to multiple MARL algorithms in FACMAC, MADDPG, QMIX, and QPLEX are demonstrated in empirical evaluations across several tasks of the SMAC, SMACv2 and GRF benchmarks. Lastly, the work shows further analysis and ablations to show the impact of each of the three novel proposed components.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback, especially for your detailed feedback on the evaluation of TEE and the impact of our proposed techniques. We were glad to hear that you found our work well-structured, clearly motivated, and systematic empirical analysis. Below, we address the key concerns you raised.\n\n> 1. I would expect clear empirical evidence that averaged TD targets and restricted action representations reduce TEE. \n\nFor a good estimation of the TD target, both bias and variance play a crucial role. The estimation error of the TD target typically consists of both components. While annealed multi-step bootstrapping reduces bias introduced by extrapolation, it increases variance by assigning higher weight to trajectory returns. Therefore, averaged TD target is importantly utilized to mitigate variance of the target Q-function, and indeed reduced it as shown in Figure 9 (left). \n\nRegarding RAR, it tackles both bias and variance of TEE by directly simplifying the joint Q-function. Intuitively, in an extreme case where the discrete action space is compressed into a single dimension, learning the joint Q-function becomes as simple as learning a V-function (though this introduces a significant optimality gap, as discussed in Section 3.1).\nTo further demonstrate the impact of RAR on TEE, we conducted experiments on MADDPG-RAR in MPE. Please refer to point 1 of our rebuttal to reviewer Lbue for details.\n\n> 2. Would the authors be able to show that AEQMIX indeed performs better than EQMIX?\n\nYes, AEQMIX consistently outperforms EQMIX. Please see point 2 of our rebuttal to reviewer Lbue for supporting results.\n\n> 3. In MADDPG-RAR, Which joint actions are mapped to the same representation and which joint actions are separated?\n\nStrictly speaking, all actions have different representation. This is because $g(a)$ is a multi-categorical distribution, and every joint action can have unique probability.\n\nFor example, consider the spread task with 5 agents and each has 5 actions, which maps from a 5^5 joint action space to 2^5. If we directly apply argmax on the probability, the joint actions are divided into 2^5 groups, each group contains between 64 and 127 mapped joint actions. Within each group, however, we did not find significant correlation among these joint actions.\n\n> 4. The $\\lambda$ values of QPLEX-RAR when applying the Sigmoid function.\n\nThe following table presents $\\lambda$ values when applying the Sigmoid function. It shows that $\\lambda$ values remain stable throughout training.\n\n| $\\lambda$\\Step|1M|2M|3M|4M|5M|6M|7M|8M|9M|10M|\n|-|-|-|-|-|-|-|-|-|-|-|\n| $\\lambda mean$|0.52|0.59|0.60|0.61|0.62|0.62|0.63|0.64|0.63|0.64|\n| $\\lambda max$|0.98|1.00|1.00|1.00|1.00|1.00| 1.00|1.00|1.00|1.00|\n| $\\lambda std$|0.16|0.15|0.16|0.17|0.17|0.17|0.17|0.18|0.17|0.18|\n\n> 5. How would AE- and RAR-extended algorithms perform in this illustration? Would the take-away still be that algorithms like MAPPO that do not rely on action-value functions are preferred for tasks with many agents or do the proposed approaches bridge that gap?\n\nOur proposed techniques improve performance, but the gains are not sufficient to match MAPPO. For instance, in the 5-agent scenario shown in Figure 1, MADDPG’s normalized return improves from -60 to -51. However, MAPPO achieves -41, maintaining a performance gap.\n\nThis suggests that while our methods enhance Q-function learning, they do not achieve the same scalability as the V-function-based on-policy methods. But this does not mean MAPPO is better in all case, off-policy methods have their own advantages, such as higher sample efficiency.\n\n> 6. Would the authors be able to compute the true proportions of extrapolated values and update the respective Figure?\n\nWe apologize for the mistake. The proportion of extrapolated values was not computed from the replay buffer. Instead, we recorded visited state-action pairs using a separate Python dictionary throughout training. As a result, Figure 2(a) reflects the true proportions.\n\n> 7. The work should attempt to make space to discuss related work and literature within the main work.\n\nThanks for the suggestion. While we included some discussion of related work in Section 2, we acknowledge that it may not be sufficient. We will consider adding a new subsection in Section 2 to better integrate the literature discussion into the main paper.\n\n> 8. The legend of Figure 10 (b) within the supplementary material is misleading.\n\nThe term \"double Q-learning\" in Figure 10(b) refers precisely to [1, 2]. In this figure, the x-axis represents different ensemble sizes, where a larger M leads to a more conservative (underestimated) target. All blue columns apply double Q-learning, which can also contribute to underestimation. To provide contrast, we include a case without double Q-learning when M=0 (pink), which exhibits the most overestimation. The figure demonstrates that simply reducing overestimation does not necessarily lead to better performance.", "claims_and_evidence": "Overall, the claims in this work are mostly well supported and convincing. However, the key claim that all three proposed techniques mitigate TEE (e.g. made in Conclusion) is insufficiently supported. The evaluation clearly demonstrates that all three proposed techniques improve the performance of off-policy MARL algorithms, but only for annealed multi-step boostrapping do the authors provide explicit analysis and show how this reduces TEE (Figure 9 left). Given the claims made, I would expect clear empirical evidence that averaged TD targets and restricted action representations reduce TEE.", "methods_and_evaluation_criteria": "The proposed techniques are sensible and clearly motivated with the problem of target estimation error and value overestimation. The evaluation is well conducted, systematic, and sufficiently extensive.", "theoretical_claims": "I went through the derivations and steps presented in the main paper and found these logical. I did not verify the proofs provided in Appendix B of the supplementary material.", "experimental_designs_or_analyses": "1. Figure 9 (left) clearly shows the impact of the proposed $\\lambda$ annealing and choice in general on the TEE. However, similar visualisation is missing for the averaged TD target computation and restricted action representation. To support the claims made in this work, I would expect to see similar visualisations to Figure 9 (left) for e.g. AQMIX/ AMADDPG with different $M$, and for MADDPG/ FACMAC/ QPLEX with and without RAR.\n2. Figure 7 (left) compares the win rates of QMIX, AEQMIX and different ablations of the AEQMIX algorithm. While EQMIX with varying $M$ all seem to significantly improve upon QMIX, no significant performance difference can be observed between EQMIX ($M=8$) and AEQMIX ($M=8$) as claimed in the text. Would the authors be able to show such ablation potentially for another task and/ or aggregated across multiple tasks to show that AEQMIX indeed performs better than EQMIX?\n3. In Section 4.3 and Section 5, empirical evidence is provided that the restricted action representation is improving the performance of the MADDPG and QPLEX algorithms. However, there is a lack of analysis to show how the RAR technique affects the target estimation error and what is being learning. In particular, I would suggest the following analysis:\n\t1. MADDPG-RAR: Would the authors be able to visualise or otherwise provide intuition into the learning low-dimensional representation of the high-dimensional action space represented by function $g$? Which joint actions are mapped to the same representation and which joint actions are separated?\n\t2. QPLEX-RAR: Figure 4 visualises the $\\lambda$ values for QPLEX. Would the authors be able to show similar values for QPLEX-RAR when applying the Sigmoid function? I know that these values would be within [0, 1] but it is not clear to me whether these would remain more stable or change similarly erratic as for QPLEX.\n\t3. Impact on target estimation error: The work claims that all three proposed techniques (including RAR) mitigate TEE, so I would expect explicit analysis showing how MADDPG-RAR and QPLEX-RAR exhibit lower TEE than their vanilla counterparts.\n4. Figure 1 nicely illustrates the challenge of learning action-value functions in MARL in tasks with many agents. How would AE- and RAR-extended algorithms perform in this illustration? Would the take-away still be that algorithms like MAPPO that do not rely on action-value functions are preferred for tasks with many agents or do the proposed approaches bridge that gap?\n5. When describing Figure 2, it is stated that the proportion of extrapolated values \"is calculated based on the fraction of $(s, a')$ pairs in each update that are absent from the replay buffer\". However, just because a state-action pair is absent from the replay buffer does not necessarily mean that it is extrapolation since it could have been trained on earlier during training (unless the replay buffer is large enough to fit all training samples). Because of this discrepancy, I would expect a lower proportion of values to actually be extrapolated than shown in Figure 2 (a). Would the authors be able to compute the true proportions and update the respective Figure?", "relation_to_broader_scientific_literature": "As stated above, I would have liked to see the discussion of related work in the main body of the paper and not as part of the supplementary material. I would encourage the authors to identify ways of fitting at least some of this within the main work.", "essential_references_not_discussed": "I am not aware of any essential references that are not discussed.", "other_strengths_and_weaknesses": "I would like to commend the author to a well structured and written paper. I enjoyed reading the work and believe that it naturally presents the problem, supports it with clear visualisations (e.g. Figure 1 and 2) to illustrate and provide evidence, provides supporting theory, and proposes conceptually simple resolutions that are grounded in existing literature.", "other_comments_or_suggestions": "I have no further suggestions or comments.", "questions_for_authors": "1. Given the claims made by this work, I would expect explicit evidence that shows how each of the proposed components (Annealed multi-step bootstrapping, averaged TD targets, restricted action representations) reduce the TEE but such evidence is only provided for annealed multi-step bootstrapping (Figure 9 left). I would strongly encourage the authors to provide such evidence for the other two techniques, and I will increase my score if this is done convincingly.\n2. Would the authors be able to visualise or otherwise provide intuition into the low-dimensional representation learned in MADDPG-RAR of the high-dimensional action space represented by function $g$? Which joint actions are mapped to the same representation and which joint actions are separated?\n3. Would the authors be able to show $\\lambda$ values for QPLEX-RAR when applying the sigmoid function as proposed, similar as shown in Figure 4 for QPLEX?\n4. Figure 1 nicely illustrates the challenge of learning action-value functions in MARL in tasks with many agents. How would AE- and RAR-extended algorithms perform in this illustration?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2rn430WV8", "reviewer": "Reviewer_Lbue", "summary": "This paper studies the extrapolation error in off-policy multi-agent reinforcement learning that is caused by the curse of multi-agent. To mitigate the errors, the paper proposed to focus on target estimation error (TEE). To further address the TEE, 3 different approaches, annealed multi-step bootstrapping, average TD target and restricted action representation (RAR), are proposed. Empirical simulations show that the proposed approaches show promising results in comparison with their vanilla counterparts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for your time and effort in reviewing our paper, which has greatly helped us improve the quality of our paper. We were glad to hear that you found our proposed methods sensible and that our empirical results demonstrated promising improvements over baseline approaches.\nBelow, we address the key concerns you raised.\n\n> 1. Some detailed explanations or analysis are needed, for example on why RAR versions are outperforming their counterparts in Figure 4-6.\n\nThe superior performance of RAR versions can be attributed to the reduction in Target Estimation Error (TEE) when the joint action space is mapped into a lower-dimensional space. To validate this, we conducted experiments with MADDPG-RAR using different joint action dimensions and report the corresponding TEE values in the following table:\n\n| Action Dim \\ Step | 0M |1M|2M|3M|4M|5M|\n| - | - | - | - | - | - | - |\n| 4  | 0.13 | 0.09| 0.09|0.07|0.05|0.04|\n| 16 | 0.17| 0.11| 0.10| 0.08|0.06|0.06|\n| 64 | 0.26| 0.10| 0.09| 0.10|0.08|0.09|\n| 256| 0.27| 0.15| 0.11| 0.09|0.09|0.08|\n|original MADDPG(3125)|0.36|0.18|0.13|0.13|0.11|0.10|\n\nFrom these results, we observe that a lower action dimension leads to a smaller TEE. This occurs because reducing the joint action space mitigates extrapolation in the target Q-function.\n Intuitively, if the joint action is mapped into a single dimension (although suboptimal due to potential bias and increased optimality difference, as discussed in Section 3.1), the target Q-function would require no extrapolation even for unseen actions. By properly constraining the joint action dimension, we can significantly reduce extrapolation while maintaining sufficient expressiveness in the joint Q-function, ultimately improving performance.\n\nAnother key intuition is that joint Q-functions do not necessarily need to assign distinct values to every possible joint action. For example, consider two agents A and B:\n\n-\tJoint action $a_1$: A moves toward B, B stays.\n-\tJoint action $a_2$: A stays, B moves toward A.\n\nIf the task depends only on the relative distance between A and B, then $a_1$ and $a_2$ may be equivalent in terms of their Q-value. Now, assume:\n\n-\t$a_1$ and $a_2$ is frequently observed in state $s_1$. \n-\t$a_1$ is frequently observed in state $s_2$, but $a_2$ is not. \n\nIn an unrestricted joint Q-function, $Q(s_2,a_2)$ may be poorly estimated due to lack of experience. In contrast, with RAR, since the model has already learned in $s_1$ that $a_1$ and $a_2$ are equivalent, and since $Q(s_2,a_1)$ is well estimated, $Q(s_2,a_2)$ will also be well estimated. This further supports why RAR enhances performance.\n\n> 2. Among the three approaches, it will be more beneficial to understand and compare the contributions of each induvial approach.\n\nFigure 7 (left) illustrates the contributions of Annealed Multi-Step Bootstrapping and Averaged TD Target to QMIX. To further clarify their impact, we provide additional results:\n\n| algorithm\\map     | zerg_5_vs_5| zerg_10_vs_10| zerg_10_vs_11| zerg_20_vs_20| protoss_5_vs_5| terran_5_vs_5| terran_10_vs_10| terran_10_vs_11|\n| - | - | - | - | - | - | - | - |-|\n|AEQMIX|62.1|64.4|47.4|56.0|78.1|76.2|80.3|68.9|\n|EQMIX|57.8|62.6|43.4|55.1|75.3|75.1|78.3|63.8|\n|QMIX|40.4|45.0|26.5|33.1|69.5|64.4|66.6|40.7|\n\nThese results indicate that both Annealed Multi-Step Bootstrapping and Averaged TD Target contribute to performance improvement. However, Averaged TD Target provides a greater benefit at the cost of additional network computations, whereas Annealed Multi-Step Bootstrapping is more computationally efficient.\n\nThe impact of RAR techniques on performance is shown in Figures 4 and 8, where we compare MADDPG and QPLEX with and without RAR. Overall, these results reinforce that all three approaches contribute to performance gains.\n\n\n\n> 3. It seems like some of the aspects studied in the paper even applies to on-policy setting in MARL, will extrapolation error also be effectively mitigated by the proposed approaches?\n\nWhile some insights from our work could extend to on-policy MARL, extrapolation error is less of a concern in such settings because on-policy methods primarily use a value function (V-function) rather than a Q-function. Specifically:\n\n- Annealed Multi-Step Bootstrapping: The on-policy equivalent is TD($\\lambda$) which is already widely used. Unlike in off-policy RL, annealing $\\lambda$ in TD($\\lambda$) is unnecessary because it does not introduce bias as in Q($\\lambda$).\n\n- Averaged TD Target: While applicable to on-policy RL, its effectiveness is reduced. The V-function is inherently easier to learn than the Q-function and is less affected by variance induced by the joint action space.\n\n- Restricted Action Representation: This technique is not applicable to on-policy RL, as it specifically targets the joint action space, which is not used in V-function-based methods.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Proposed methods make sense.", "theoretical_claims": "The reviewer didn’t check the correctness of the proofs.", "experimental_designs_or_analyses": "The reviewer didn’t implement the pseudo code in the local machine to redo the experimental studies, but went over the simulation results.", "relation_to_broader_scientific_literature": "The paper proposed 3 different approaches in addressing the large action space issue in multi-agent setting. These approaches, notably including restricted action representation, might provide inspirations in large action space problems.", "essential_references_not_discussed": "The reviewer believes related works are sufficient.", "other_strengths_and_weaknesses": "Strength: The paper studied the large action space issues in multi-agent system. By focusing on TEE, the paper proposed 3 different approaches, among which RAR is the most interesting to the reviewer. The empirical results show promising results for these approaches in comparison with their vanilla counterparts.\nWeakness: Please see the comments sections.", "other_comments_or_suggestions": "1.\tSome detailed explanations or analysis are needed, for example on why RAR versions are outperforming their counterparts in Figure 4-6.\n2.\tAmong the three approaches, it will be more beneficial to understand and compare the contributions of each induvial approach.\n3.\tIt seems like some of the aspects studied in the paper even applies to on-policy setting in MARL, will extrapolation error also be effectively mitigated by the proposed approaches?", "questions_for_authors": "Please see the comment section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles value target estimation errors in off-policy MARL, showing how the large joint action space leads to significant estimation issues. It decomposes value error into three components, highlighting the impact of target estimation error, particularly in value decomposition methods. To address this, the authors propose: (1) multi-step target estimation, (2) ensemble averaging of target values, and (3) projecting actions into a lower-dimensional discrete space. These techniques are evaluated on FACMAC, MADDPG, QMIX, and QPLEX across the SMAC, SMACv2, and GRF benchmarks, with ablation studies. This is a very nice work and all reviewers unanimously recommended for acceptance.
Decision: Accept (poster)