Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
Authors: Jianzhun Shao
Yun Qu
Chen Chen
Hongchang Zhang
Xiangyang Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-agent reinforcement learning
offline reinforcement learning
TL;DR: We introduce a novel multi-agent adaptation of conservative Q-learning, advancing the state-of-the-art in offline MARL.

Abstract: Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. To mitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent conservative methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
Jianzhun Shao, Yun Qu Chen Chen, Hongchang Zhang, Xiangyang
Department of Automation
Tsinghua University, Beijing, China
{ajz18, qy22, hc-zhang 1aizhang199)mmm Amaiils. l tsinghua.edd.
cclvr0163. com xyji@tsinghin edu.
Abstract
Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. To mitigate this prob- lem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL)
conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent methods it, CFCQL calculates conservative regularization for each agent separately in counterfactual way and then linearly combines them realize an overall conservative value estimation. We prove that still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improve- ment bound are independent of the agent number, which is therefore theoretically superior the direct treatment referred above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.
1 Introduction
Online Reinforcement Learning (Online RL) needs frequently deploying untested policies to environ- ment for data collection and policy optimization, making it dangerous and incfficient to apply in the real-world scenarios (e.g. autonomous vehicle teams). While, Offline Reinforcement Learning (Of- fline RL) aims to learn policies from a fixed dataset rather than from interacting with the environment, and therefore is suitable for the real applications with highly safety requirements without efficient simulators (25].
Directly applying off-policy RL to the offline setting may fail due to overestimation [3] 24 Existing works usually tackle this problem by pessimism. They either utilize behavior regularization to constrain the learning policy close the behavior policy induced by the dataset [49 20 [1]
conduct conservativo(pessiiittie value iteration to mitigate unexpected overestimation [18 21].
has been demonstrated both theoretically and empirically that these methods can achieve comparable performance to their online counterparts under some conditions [21)
'Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Though cooperative Multi-agent Reinforcement Learning (MARL) has gained extraordinary success in various multiplayer video games such as Dota 4 StarCraft 440 and soccer [3], applying current MARL in real scenarios is still challenging due to the same safety and efficiency concerns in single- agent setting, then it worth conducting investigation for offline RI. in multi-agent setting. Compared with single-agent setting, offline RL in multi-agent setting has its own difficulty. On the one hand, the action space explodes exponentially as the agent number increases, then an arbitrary joint action is more likely to be an OOD action given the fixed dataset size, making the OOD phenomenon more severe. This will further exacerbate the issues of extrapolation error and overestimation in the policy evaluation process and thus induce an unexpected or even disastrous final policy [52]. On the other hand, as the core of MARL, the agents need to consider not only their own actions but also other agents' actions well as contributions to the global return in order achieve overall high performance, which undoubtedly increases the difficulty for theoretical analysis. Besides, instead of just guaranteeing single policy improvement, the bounded team performance is also key concern to offline MARI..
There exist few works combine MARL with offline RL. Pan et al. [34] uses Independent Learn- ing [44) as a simple solution, i.c., each agent regards others as part of the environment and inde- pendently performs offline RL learning. Although mitigating the joint action OOD issue through decoupling the agents's 's learning completely, it essentially still adopts a single-agent paradigm to learn and thus cannot enjoy the recent progress MARL that centralized value function can empower better team coordination 37 27]. To utilize the advantage of centralized training with decentralized exccution (CTDE), Yang al. [52 applies implicit constraint approach the value decomposition network 43 which alleviates the extrapolation error and gains some performance improvement empirically. Although it proposes some theoretical analysis for the convergence property of value function, whether the team performance can be bounded from below as agents number increases remains still unknown.
In this paper, we introduce a novel offline MARL algorithm called Counterfactual Conservative Q-Leaming (CFCQL) and aim to address the overestimation issue rooted from joint action OOD phenomenon. It adopts CTDE paradigm to realize team coordination, and incorporates the sfate-of- the-art offline RL algorithm CQL to conduct conservative value estimation. CQL is preferred due to its theoretical guarantee and flexibility of implementation. One direct treatment is to regard all the agents as a single one and conduct standard CQL learning on the joint policy space which we call MACQL. However, too much conservatism will be generated in this way since the induced penalty can be exponentially large in the joint policy space. Instead, CFCQL separately regularizes each agent in counterfactual way to avoid too much conservatism. Specifically, each agent separately contributes CQL regularization for the global Q value and then a weighted average is used as an overall regularization. When calculating agent i's regularization term, rather than sampling OOD actions from the joint action space as MACQL does, we only sample OOD actions for agent and leave other agents' actions sampled in the dataset. We prove that CFCQL enjoys underestimation property as CQL does and the safe policy improvement guarantee independent of the agents number n, which is advantageous to MACQL especially under the situation with a large n.
We conduct experiments 1 man-made environment Equal Line, and 3 commonly used multi-agent environments: StarCraft II [40), Multi-agent Particle Environment (27), and Multi-agent MuJoCo [5), including both discrete and continuous action space setting. With datasets collected by Pan al. 34 and ourselves, our method outperforms existing methods in most settings and even with large margin on some of them.
We summarize our contributions as follows: (1) we propose a novel offline MARL method CFCQL based on CTDE paradigm to address the overestimation issue and achieve team coordination the same time. (2) We theoretically compare CFCQL and MACQL to show that CFCQL is advantagous to MACQL on the performance bounds and safe policy improvement guarantee agent number is large. (3) In hard multi-agent offline tasks with both discrete and continuous action space, our method shows superior performance to the state-of-the-art.
2 Related Works
Offline RI. Standard RL. algorithms are especially prone to fail due to erroneous value overestimation induced by the distributional shift between the datasct and the learning policy. Theoretically, it is

---

proved that pessimism can alleviate overestimation effectively and achieve good performance even with non-penn con-perfect data coverage [6 22 39
53 6 In the algorithmic line, there are broadly two categories: uncertainty based ones and behavior regularization based ones. Uncertainty based approaches attempt to estimate the epistemic uncertainty of Q-values or dynamics, and then utilize this uncertainty to pessimistically estimating Q in model-free manner 2 50) 3 or conduct learning on the pessimistic dynamic model in a model-based manner [6]. Behavior regularization based algorithms constrain the learned policy to lie close to the behavior policy in either explicit implicit 18 II and is advantageous over uncertainty based methods in computation efficiency and memory consumption. Among these class of algorithms, CQLÍ11 is preferred due to its superior empirical performance and flexibility of implementation.
51
ways 20 13 49
MARL. A popular paradigm for multi- agent reinforcement learning is centralized training with decentralized execution (CTDE) 37 27]. Centralized training inherits the idea of joint action learning 77. empowering the agents with better team coordination. And decentralized execution enjoys the deploying flexibility of independent learning 44 CTDE, some value-based works concentrate on decomposing the single team reward to all agents by value function factorization [F 37., based on which they further derive and extend the Individual-GGoballaal for policy optimality analysis 442 framework, using a centralized critic to guide each agent's 's policy update [27 Some variants of
(IGM) principle 46.. Another group of works focus the actor-critic
47
38
PPO [41,, including IPPO (8), MAPPO [53], and HAPPO [19] also show great potential in solving complex tasks. All these works use the online learning paradigm, therefore disturbed by extrapolation error when transferred to the offline setting.
Offline MARL. To make MARL applied in the more practical scenario with safety and training efficiency concerns, offline MARL is proposed. Jiang & Lu [14) shows the challenge of applying BCQ [33 to independent learning, and Pan et al. [34] uses zeroth-order optimization for better coordination among agents' policies. Although it empirically shows fast convergence rate, the policies trained by independent learning have no theoretical guarantee for the team performance. With CTDE paradigm, Yang et al. [52) adopts the same treatment as Peng et al. (36 and Nair et al.
30 to avoid sampling new actions from current policy, and Tseng et al. 455 regards the offline MARL sequence modeling problem, solving it by supervised learning. As result, both methods' performance relies heavily on the data quality. In contrast, CFCQL does not require the learning policies stay close to the behavior policy, and therefore performs well datasets with low quality. Meanwhile, CFCQL is theoretically guaranteed to be safely improved, which ensures its performance on datasets with high quality.
3 Preliminary
3.1 MARL Symbols
We use a decentralised partially observable Markov decision process (Dec-POMDP) G (S. A. I, P,P, Z, O, n. ?)) to model fully collaborative multi-agent task with n agents, where E S the global state. At time step t. each agent I {L..., n) chooses an action a' E A, forming the joint action a E A = A". T(s' s, a) S x A x S + [0. 1] is the environment's 's state transition distribution. All agents share the same reward function r(s. a)
32
x A - R. 7 E [0, 1) is the discount factor. Each agent i has its local observations of E O drawn from the observation function
Z (s, i) : 5 x I + O and chooses an action by its policy (aaloo'') 0 A[0,, 1||41). ). The agents' joint policy π : [T= T² induces joint action-value function: (T a) [[[(|s, a), where R Etz 7'r, is the discounted accumulated team reward. We assume Vr, |r| Rmax- The goal of MARL is to find the optimal joint policy π² such that Q (s, a) 2 (*,(sa)), for all π and (s, a) E S x A. In offline MARL, we need to learn the optimal π from a fixed dataset sampled by an unknown behaviour policy B3, and we can not deploy any new policy to the environment to get feedback.
3.2 Value Functions in MARL
In MARL settings with discrete action space, we can maintain for each agent local-observation- based Q function Q'(o", a²), and define the local policy x² = arg max, Q'(ot,a). To train local policies with single team reward, Rashid et al. [3] proposes QMIX, using global Qiot to model
3

---

the cumulative team reward. Define the Bellman operator T as TQ(s. =r+ max Q a'). QMIX aims to minimize the temporal difference:
Q f- arg min E, -. - (s,
where 7 is the empirical Bellman operator using samples from the replay buffer D, and Qiat is the empirical global Q function, computed from s and Q's represented by neural network satisfying aQ jaQ' 2 O. With such restriction, QMIX can ensure the arg max Qa (s. a) (arg m Q' (o' ()), .... arg m Q" (o", a²)). For continuous action space, neural network is used to represent the local policy x' for each agent. Lowe et al. [27] proposes MADDPG, maintaining a centralized critic Q to directly guide the local policy update
principle that
by gradient descent. And the update rule of Qiot is similar to Eq. just replacing T with T
==Q=r+yP* Q, where pd ((x a) a"). 1Q aa))
3.3 Conservative Q-Learning
Offline RL is prone to the overestimation of value functions. For an intuitive explanation of the under- lying cause, please refer to Appendix E.1. Kumar et al. [2]] proposes Conservative Q-Leaming (CQL), adding a regularizer to the Q function to address the overestimation issue, which maximizes the Q function of the state-action pairs from the dataset distribution B, while penalizing the function sampled from new distribution (e.g., current policy πτ. The conservative policy evaluation is shown as follows:
Qu+1 f arg min alEaa (aw.://wowiiiss.------e-eeaedl a)]] + B
-T" a))
As shown in Theorem 3.2 in Kumar et al. (1]. with a large enough a, we can obtain a Q function, whose expectation over actions is lower than that of the real Q, then the overestimation issue can be mitigated.
Proposed Method
We first show the overestimation problem of multi-agent value functions in Sec. 4.1 and di- rect solution by the naive extension of CQL to multi-agent settings in Sec. 4.2 which we call MACQL. In Sec. 4.3 propose Counterfactual Conservative Q-Learning (CFCQL) and further demonstrate that it can realize conservatism in a more mild and controllable way which is inde- pendent of the number of agents. Then we compare MACQL and CFCQL from the aspect of safe improvement performance. In Sec. 4.4, we present our novel and practical offline RL algorithm.
4.1 Overestimation in Offline MARL
MACOL
CMX
CPCOL
The Q-values in online learning usually face the overestimation problem I 33), which becomes more severe in offline settings due distribution shift and offline policy evalu- ation 133 Offfine MARL suffers from the same issue even worse since the joint action space explodes exponentially as the agent number increases, then an arbitrary joint ac- tion is more likely to be an OOD action given the fixed dataset size. To illustrate this phenomenon, we design a toy Multi- Agent Markov Decision Process (MMDP) as shown in aa). There are five agents and three states. All agents are randomly ini- tialized and need learn to take three actions (stay, move up, move down) to move to or stay in the state S2. The dataset consists of 1000 samples with reward larger than 0r As shown in Fig. [b), if not specially designed for offline setting,
(a)
(c)
Figure 1: (a) decomposable Multi-Agent MDP
which urges the agents moves to or stays in S2. The number of agents is set to 5. (b) The learning curve of the estimated joint state-action value function in the given MMDP. The value is indicated by the dotted line, which represents the maximum discounted return. (c) The performance curve of the corresponding methods.

---

naive application of QMIX leads to exponentially explosion of value estimates and the sub-optimal policy performance, Fig. (c).
4.2 Multi-Agent Conservative Q-Learning
When treating multiple agents as unified single agent and only regarding the joint policy of the
G, jr, R, that is ((a s) Ih (a*)s),, [ II (²(aals) and ((aas)) = -Ii-, (s), we can extend the single-agent CQL to CTDE multi-agent setting straightforwardly and derive the conservative policy evaluation style directly as follows:
Qtat < arg min ES-D.a-
Ea- + (D(
(3)
where we represent the temporal difference (TD) error concisely ED(T,, Q,
s''.a.'' (Q(s, a) f Q (s. a)) In the rest of the paper, we mainly focus on the global Q
function of a centralized critic and omit the superscript "tot" for ease of expression.
Similar CQL, MACQL can lear lower-bounded Q-value with a proper a, then the overestimation issue in offiine MARL can be mitigated. According to Theorem 3.2 in Kumar et al. (2T). the degree
3 I]. However, as will show in Sec. Door expands exponentially the number of agents increases, resulting in an over-pessimistic value function and a mediocre policy improvement guarantee for MACQI..
of pessimism relies heavily on DcQu ( (T, (s) := Ea (aass)
4.3 Counterfactual Conservative Q-Learning
Intuitively, aim to learn value function that prevents the overestimation of the policy value, which is expected not too far away from the true value in the meantime. Similar to MACQL as introduced above, we add a regularization in the policy evaluation process, by penalizing the values for OOD actions and rewarding the values for in-distribution actions. However, rather than regarding all the agents single one and conducting standard CQL method in the joint policy space, in CFCQL method, each agent separately contributes regularization for the global Q value and then a weighted average is used as overall regularization. When calculating agent i's regularization term, instead of sampling OOD actions from the joint action space as MACQL does, we only sample OOD actions for agent and leave other agents' actions sampled in the dataset. Specifically, using a - to denote the joint actions other than agent i. Eq, is our proposed policy evaluation iteration:
n
Qu+1 < argmin min
A,E D. t- Σ
(Q(s, a)] t-.
B a)) + E Q.K.,
(4)
where at Aj are tlyer-parameett and Σ A; = 1, 0. We refer to our method as 'counterfac-
tual' because its structure bears resemblance to counterfactual methods in MARL [9]. This involves obtaining each agent's 's counterfactual baseline by marginalizing out single agent's action while keeping the other agents' actions fixed. The intuitive rationale behind employing counterfactual-Iin approach is that by individually penalizing each agent's di-of-ddittibuutinn (OOD) actions while hold- ing the others' actions constant from the datasets, can effectively mitigate the cattof-distribution problem in offline MARL with reduced pessimism, as illustrated in the rest of this section.
The theoretical analysis is arranged as follows: In Theorem 4.1 leads to a milder conservatism on value function, which still lower bounds the true value. Then we compare the conservatism degree between CFCQL and MACQL in Theorem 4.2 In Theorem 4.3 and Theorem 4.4 we show the effect of milder conservatism brought by counterfactual treatments on the performance guarantee (All proofs are defered to Appendix A).
we show the new policy evaluation
Theorem 4.1 (Equation Daautiionnresun results in a lower bound of value function).
The value of the policy under the Q function from Equation 4 1.m (s) = E (ale) (QT (s, a)), lower-bounds the true value of the policy obtained via exact policy evaluation, (T) (s)= = Err(als) 1QT (TT(( (s, a)), when μ n,, according to: Vs E
Cr.TT, Raax D. D (s) < V* (s) a [T ) E Σn d z - =1] (s) + (T (T-pPm)-
(1 (1-7) P
Define Ceq (3)) == Ea r(als) \ 1
(at1e)
- 11. If Cr.. max,eD (P((s) 1-y
1
(i(a))))
5

---

CQL (T. ())))(s) vs D, (8) S V (T)s,, with probability 2 1 - 8. When T T", then any a > 0 guarantees (s))s S V (s), Vs E
Theorem mainly differs from Theorem 3.2 in Kumar et al. 21 on the specific form of the divergence: DEoL and DEG both of which determine the degrees of conservatism when я and ß different. Empirically, we find DeqL generally becomes too large when the number of agents expands, resulting in a low, even negative value function. To measure the scale difference between (Doqu(T', (8) + +1-111 and DE ()()) E; A;Doqu (x', (8))
DoQu( (s)
we have Theorem
Theorem 4.2. Vs, n, G,, one has S D (T, < DoQu(TT (T, and the following in- equality holds:
DooL n ()) CQL ( ())
2 exp
Σ i=1ii
KL(T(+)118())
(5)
(a) where j = arg max; Egt (*()) represents the agent whose policy distribution is the most far away from the dataset.
Since DE6 can be written as E, X, A.Don , (8), can be regarded as a weighted average of each individual agents' policy deviations from its individual behavior policy. Therefore, the scale of DCR is independent of the number of agents. Instead, Theorem 4.2 shows that both Deqr and the CQL
ratio of DeQL and DEQ explode exponentially as the number of agents increases.
Then we discuss the influence of DEE, on the property of safe policy improvement. Define M as the MDP induced by the transitions observed in the dataset. Let the empirical return (TT, M) be the discounted return for any policy in M and Q be the fixed point of Equation Then the optimal policy of CFCQL is equivalently obtained by solving:
1
(als) + arg max (T, M) a Exwdy (1) (TL ())].
DD ()))))
(6)
See the proof in Appendix A.3. Eq. 6 can be regarded as adding a CQL -elatel regularizer to J(,, M. Replacing DeC with DeQL in the regularizer, we can get a similar form of TM for MACQL. We next diseuss the performance bound of CFCQL and MACQL on the true MDP M. If we assume the full coverage of ( on M, we have:
Theorem 4.3. Assume Vs, ds,a,1, 1, (a)s) 2 e, then with probability 2
J(TMA M) 2 J(TT, (M) a
7 en - -1) -smpliig error,
J(TÖF, M) J( M)
-1) sampling error,
where n² = arg max (T, M), i.e. the optimal policy, and sampling error is a constant dependent on the MDP itself and D.
Theorem 4.3 shows that when sampling error and a are small enough, the performances gap induced by FF can small, but that induced by Ti expands when increases, making л MM far away from the optimal. Upon examining the performance gap, we ultimately compare the safe policy improvement guarantees of the two methods on M:
Theorem 4.4. Assume Vs, a. 3 (als) 2 c. The policy TMA (als) is ( MAA policy improvement over in the actual MDP M. i.e., J. 2 J(B.M { MAA.A. And the policy (a)s)))) is CCF-safe -safe policy improvement over B in M, i.e., J( M) 2 J(B CFF When
> log! (: (1+2
VIA [[D()
(Cr. + RRmaCCT. 1-y
(2-1) OF MA
Detailed formation of MMA and CCF is provided in Appendix A.5. Theorem 4.4 shows that with large enough agent count n. CFCQL has better safe policy improvement guarantee than MACQL.. The validation experiment of this theoretical result is presented in Section 5.1

---

4.4 Practical Algorithm
The fixed point of Eq. provides an underestimated Q function any policy μ. But it is computationally inefficient to solve Eq. - every time after one step policy update. Similar to CQL [21]. we also choose ((aas)) that would maximize the current Q with p-regularizer. the regularizer aims to minimize the KL divergence between je and a uniform distribution, (aa)ss & exp (Ea--~3-.((5- a', a=)), which results in the update rule of Eq. 7 (See Ap-
pendix B.1 for detailed derivation):
min &B
Σ A. - (loggllo
Σ exp(Q(ss, a))] Eans Q( a)) + ED((T, Q).
(7)
Finally, we need to specify each agent's 's weight of minimizing the policy Q function X; Theoretically. any simplex of À that satisfies Σ X; = can be used induce an underestimated value function linearly increasing as agents number as we expect. Therefore, simple way to set X; n Vi where each agent contributes penalty equally. Another way is to prioritize penalizing the agent that exhibits the greatest deviation from the dataset, which is the one-hot style of A:
{
1.0, i arg max, Bags x'(s) others
(s(s)
(P]())
0.0,
We assert that each agent's conservative contribution deserves to be considered and differed according their degree of deviation. As a result, both the uniform and the one-hot treatment present some limitations. Consequently, we employ moderate softmax variant of Eq.
exp(rTE x ('())
si,s,((s)) Σ/= ( 3T
3(3)
(9)
A When + 0, A; + 1, and when T - 0o. it turns into Eq. To compute Eq. 9 we need an explicit expression of o² and St. In discrete action space, can be estimated by exp Q(s, a', t)), and we use behavior cloning [29 train parameterized ((s) from the dataset.
where is predefined temperature coefficient, controlling the influence of E 37
continuous action space, =" is parameterized by each agent's local policy. For Sr, we use the method of explicit estimation of behavior density in Wu et al. [48], which modified from a VAE [17 estimator. Details for computing are defered to Appendix B.2.
For policy improvement in continu- ous action space, we also take deriva- tion of counterfactual function for each agent, rather than updating all agents' policy together like in MAD- DPG. Specifically, the gradient of each agent i's policy Æ² is calculated by:
Algorithm CFCQL-D and CFCQL-C
1: Initialize Qo, target network Q target update interval fear. replay buffer D, and optionally T& for CFCQL-C
2: for = 2, tmaz do
3: Sample transitions {8, a, sr, from D
4: Compute (o a) using the structure of QMIX for CFCQL-D or MADDPG for CFCQL-C.
5: Calculate À according to Eq.
V.r T..Bg.--iD,e. a' Qo(s, a)
6: Update Qu by Eq. with sampled transitions. Using
(10) The reason is that in CFCQL, only minimize Q(s, T², 3B- rather than ((s,T) T). Using the untrained Q(s, , T) to directly guide like MADDPG may result in bad policy.
Ta for CFCQL-D, and 7/ for CFCQL-C
7: (Only for CFCQL-C) For each agent take one-step policy improvement for म according to Eq.
8: if mod tar then
9: update target network i <-
10: end if
We summarize CFCQL in discrete and continuous action space in Algo- rithm | as CFCQL-D and -C, sepa- rately.
II: end for

---

Experiments
Baselines. We compare our method CFCQL with several offline Multi-Agent baselines, where baselines with prefix M A adopt CTDE paradigm and the others adopt independent learning paradigm: BC: Behavior cloning. TD3-BC[IT): One of the state-of of- single agent offline algorithm, simply adding the BC term TD3 [12]. MACQL: Naive extension of conservative Q-learning, proposed in 4.2 MAICQ[22: Multi-agent version of implicit constraint Q-learning by decomposed multi-agent joint-poliey under implicit constraint. OMAR 34 Using zeroth-order optimization for better coordination among agents' policies, based on independent CQL (ICQL). MADTKD Using decision transformer to represent each agent's policy, trained with knowledge distillation. [Q] and :WAC[3]]: variants of advantage weighted behaviour cloning, which are SOTA on single agent offline RL. Details for baseline implementations are Appendix C.3.
Each algorithm run for five random seeds, and report the mean performance with standard deviation Four environments are adopted to evaluate method, including both discrete action space and continuous action space. We have relocated additional experimental results to Appendix to conserve space.
macql
Number of agents (b)
(a)
5.1 Equal Line
Figure (a) The Equal_Tive environment where
To empirically compare CFCQL and MACQL.
n=3. (b) Performance ratio of CFCQL and MACQL to the behaviour policy with a varing
agents number increases, design multi-agent task called Equal__iite Line, which one-dimensional simplified version of Equal_Space introduced Tseng et al. [45] Details of the environment ensure every agent is equally spaced. The datasets consist of 1000 trajectories sampled by executing a pulla-pretrained policy of QMIX(37). i.e. Expert dataset. We plot the performance ratios of CFCQL and MACQL to the behavior policy for different agent number n in (b). It can be observed that the performance of MACQL degrades dramatically the increase of number of agents while the performance of CFCQL remains basically stable. The results strongly verify the conclusion we proposed in Sec[4.3] that CFCQL has better policy improvement guarantee than MACQL with large enough number of agents n.
number of agents.
in Appendix C.I. The agents need cooperate to disperse and
5.2 Multi-agent Particle Environment
In this section, we test CFCQL Multi-agent Par- ticle Environemnt with con- tinuous action space. We use the dataset and the adver- sary agent provided by Pan et al. (34] The performance of the trained model is mea-
Table 1: Results on Multi-agent Particle Environment. CN: Coopera- tive Navigation. PP.Predator-prey World: World.
Env
Dataset
OMAR
MACQL
IQL
CFCQL
Random 34.4+5.3
45.6+8.7
5.5+1.1
62.248.1
Med-Rep
37.9=12.3
25.5=5.9
10.8+4.5
52.2+9.6
CN
Medium
47.9=18.9
14.3+20.2
28.2+3.9
65.0+10.2
sured by the normalized score 100 (S- SRandom)/ SExpert)
Expert 114.9+2.6
12.2431
103.7+2.5
11244
Random 11.142.8
25.2+11.5
1.341.6
78.5+15.6
10
Med-Rep 47.1+15.3
11,919.2
23.2+11
71.116
In Table I only show the comparison of our method and the current state-of-the- art method OMAR and IQL save space. For plete comparison with more baselines. e.g., TD3+BC and
PP
Medium
66.7+23.2
55+43.2
53,6219.9
68.5+21.8
Expert 116.2+19.8
108.4+21.5
109.3+10.1
118.2+13.1
Random
5.9+5.2
11.7+11
2.9+4.0
68+20.8
Med-Rep
42.9+19.5
13.2+16.2
41.529.5
73.4+23.2
World
Medium
74.6+11.5
67.4248.4
70.5+15.3
93.8431.8
Expert
110.4425.7
99.7431
107.8
119.7426.4
OOr code and datasets available https://github. http://hhhw..ccmtttttaaa////////

---

AWAC, please refer to Appendx D.1. It can be seen that on 11 of the 12 datasets, CFCQL shows superior performance than current state-of-the-art Note that we only report the results of 7 In Appendix D.2 of the ablation on 7, we show that with carefully fine-tuned higher scores of CFCQL. can be obtained. We also carry out ablations of a in Appendix D.3.
5.3 Multi-agent MuJoCo
In this section we investigate the effect of our method on more complex continuous control task. We use the HalfChectah-v2 setting from the multi-agent MuJoCo en-
Table 2: Results on MaMuJoCo.
Dataset
Random
Med-rep
Medium
Expert
vironment (35) and the datasets provided
ICQ
7.4+0.0
35.6+2.7
73.6+5.0
110.6*3.3
by Pan et al. (4 Table - shows that
TD3+BC
7.4+0.0
27.1+5.5
75.5+3.7
114.4+3.8
CFCQL exceeds the current ofateoofthhe- art on most datasets.
ICQL
7.4=0.0
41.2+10.1
50.4+10.8
64,2+24,9
OMAR
13.5+7.0
57.745.1
80.4+10.2
113.5+4.3
MACQL
5.3+0.5
37.0+7.1
51.5+26.7
50.1420.1
Except for the counterfactual Q function, we also analyze whether the counterfac- tual treatment in CFCQL can be incor- porated in other components and help further improvement in Appendix D.4.
IQL
7.4+0.0
58.8+6.8
81.3+3.7
115.6+4.2
AWAC
7.3+0.0
30.9+1.6
71.214.2
113.3*4.1
CFCQL
39.724.00
59.5+8.2
80,519.6
118.5#4,9
We find that the counterfactual policy improvement, i.e., the policy improvement by Eq. 10 rather than using MADDPG's s PI, is critical for method.
5.4 StarCraft II
Table 3: Averaged test winning rate of CFCQL and baselines in StarCraft II micromanagement tasks.
Map
Dataset
CFCQL
MACQL
MAICQ
OMAR
MADTKD
BC
IQL
AWAC
medium
0.40:00110
0.17:0.88
0.18:0.22
0.15:0.04
0.18#0.03
0.1640.07
0.1640.04
0.19:0.05
medium_repla
0.55+0.00
0. 0.12+0.88
0.41=0.6
0.24=0.09
0.36+0.07
0.33+0.04
0.33+0.06
0.39+0.05
expert
0.99:0.01
0.58+0.34
0.93:0.44
0.95=0.04
0.99:0.02
0,9840.03
0.9740.03
mixed
0.84+0.09
0.67+0.17
0.85=0.07
0.60=0.04
0.47+0.08
0.44+0.06
0.19+0.04
0,1400.04
medium
0.28:0.03
0.0920.06
0.03:0.01
0.0020.00
0.0120.01
0.0820.02
0.2020.05
0.1920.03
medium_replay
0.12:0.04
0.0110.01
0.01:0.02
0.0020.00
0.01:0.01
0.0140.01
0.0420.04
0.0840..55
35_ _s
expert
0,9920.01
0.92:0.05
0.91=0.04
0.6420.08
0.67
0.9820.02
0.9920.01
0,9920.02
mixed
0.6020.14
0.1720.10
0.10:0.04
0.00g0.00
0.14:0.08
0.2140.04
0,20a0.06
0.1840.03
medium
0.2920.05
0.0110.01
0.2620.03
0.1920.06
0.2120.04
0.28a0.37
0.25a0.02
0.22a0.04
medium_ replay
0.22:0.06
0.16:0.08
0.18:0004
0.03:0.02
0.16:
0.1820.06
0.1820.04
0.1840.04
5m 6m
expert
0.84+0.03
0.01#0.11
0.72+0.05
0.33+0.06
0.58+0.07
0.82+0.04
0.77+0.03
0.75+0.02
mixed
0.76:0.07
0.0120.01
0.6720.08
0.1020.10
0.21:0.05
0.2120.12
0.7620.06
0.78+0.02
medium
0.41+0004
0.01#0.01
0.19:0.04
0.04=0.03
0.22+0.07
0.40=0.03 03
0.40#0.05
0.43+0.06
medium replay
0.21=0.05
0.08=0.04
0.07=0.04
0.00=0.00
0.12+0.05
0.11
0.17+0.03
0.1400.04
vs
expert
0.7+0.06
0.00=0.00
0.24=0.08
0.01=0.01
0.48=0.08
0.60=0.04
0.67=0.03
0.67=0.03
0.49:00.8
0.01+0.01
0.05+0.03
0.00=0.00
0.27+0.06
0.36+0.05
0.35+0.06
We further validate CFCQL's univer- sality through complex experiments on the StarCraft II Micromanage- ment Benchmark [40), encompassing four maps with varying agent counts and difficulties: 2s37, 3s_vs_57,
Test WWiretin
5m_ vs 6m, and 6h_vs_ vs. 8z. As no pre-existing datasets exist for these tasks, we generate them ourselves, with dataset creation details provided in the Appendix C.2.
0.1 02 05
10
811 0.2
05 1 10 8.1 0.2 05 10 01 02 05 10
Temperature Coefficient/ /
(a) Med-Replay
(b) Medium
(c) Mixed
(d) Expert
Table B presents the average test win- ning rates of various algorithms on different datasets. MACQL's perfor- mance depends on agent count and environmental difficulty, only succeeding in 2x3z and 3$_vs_57 MAICQ and OMAR perform well across many datasets but can not tackle all tasks and significantly
Figure 3: Hyperparameters examination on the temporature
coefficient in different types of datasets.

---

struggle in 6h vs 8z, highly challenging map. MADTKD, employing supervised learning and knowledge distillation, works well but seldom surpasses BC. IQL and AWAC are competitive base- lines but they still fall short compared to CFCQI. in most datasets. CFCQL significantly outperfoms all baselines on most datasets, achieving wate-of-the-ar results, with its success attributed to moderate and appropriate conservatism compared to MACQL. and other baselines.
Temperature Coeflicient. To study the effect of temperature coefficient т, the key hyperparameter in computing the A; Fig a(a) shows the testing winning rate of CFCQL with different 7s on each kind of dataset of map 6h vs 8z. As shown, CFCQL not sensitive to this hyperparameter and we find that the best value of r is usually greater than while lower than infinity, showing that moderate imbalance can be more effective as expected in previous section.
Discussion and Conclusion
6.1 Broader Impacts
Our proposed method holds potential for application in real-world multi-agent systems, such as intelligent warehouse management or medical treatment. However, directly implementing the derived policy might entail risks due to the domain gap between the training virtual datasets and real-world scenarios. To mitigate potential hazards, it is crucial for practitioners to operate the policy under human supervision, ensuring that undesirable outcomes are avoided by limiting the available options.
6.2 Limitations
Here we discuss some limitations about CFCQL. In the case of discrete action space, since CFCQL uses QMIX as the backbone, it inherits the Individuallglobal-man principle [42) which means it cannot solve tasks that are not factorizable. On continuous action space, the counterfactual policy update used in CFCQL allows for updating only one agent's policy for each sample, which may lead to lower convergence speed compared to methods with independent learning.
6.3 Conclusion
In this paper, we study the offline MARL problem which is practical and challenging but lack of enough attention. We demonstrate from theories and experiments that naively extending conservative offline RL algorithm to multi-agent setting leads to over-pessimism as the exponentially explosion of action space with the increasing number of agents which hurts the performance. To address this issue, we propose a counterfactual way to make conservative value estimation while maintaining the CTDE paradigm. It is theoretically proved that the proposed CFCQL enjoys performance guarantees independent of number of agents, which can avoid overpessimism caused by the exponentially growth of joint policy space. The results of experiments with discrete continuous action space show that our method achieve superior performance to the ofate-of-the-art. Some ablation study is also made to propose further understanding of CFCQL. The idea of counterfactual treatment may also be incorporated with other offline RL and multi-agent RL algorithms, which deserves further investigation both theoretically and empirically.
Acknowledgement
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801.
References
[1] Ackermann, J., Gabler, V., Osa, T., and Sugiyama, M. Reducing overestimation bias in multi- agent domains using double centralized critics. arXiv preprint wr..iv:190../// 2019.
[2] Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective offline rein- forcement learning. International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
10

---

[3] An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty-basee offline reinforcement learning with diversified g-ensemble. Advances in Neural Information Processing Systems, 34, 2021.
[4] Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint www.i: 066800 2019.
[5] Buckman, J., Gelada, C., and Bellemare, M. G. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
[6] Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint urXiv:2202.024400 2022.
[7] Claus, C. and Boutilier, C. The dynamies of reinforcement learning in cooperative multiagent systems. In AAAI Conference on Artificial Intelligence (AAAI), 1998.
[8] de Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H., Sun, M., and Whiteson,
S. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
[9] Foerster, J., Farquhar. G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAT), 2018.
[10] Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.00 072199, 2020.
[11] Fujimoto, S. and Gu, S. minimalist approach offline reinforcement learning, Advances in neural information processing systems, 34:20132-20145, 2021.
[12] Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
[13] Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforeement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
[14] Jiang, J. and Lu, Z. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2/08.0/ 01832, 2021.
[15] Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
[16] Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning, arXiv preprint uxXiv:2005.05551, 2020.
[17] Kingma, D. P. and Welling, M. 1312.613 2013.
Auto-encoding variational bayes. arXiv preprint
[18] Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 1774-5783. PMLR, 2021.
[19] Kuba, J. G., Chen, R., Wen, M., Wen, Y., Sun, F., Wang. J., and Yang. Y. Trust region policy optimisation in multi-agent reinforcement learning, arXiv preprint arXiv:2109. 112119911125 11251, 2021.
[20] Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
[21] Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative g-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1179-1191 1191, 2020.
[22] Kumar, A., Hong, J., Singh, A., and Levine, S. Should run offline reinforcement learning or behavioral cloning? In Deep RL Workshop NeurIPS 2021, 2021.
[23] Kurach, K., Raichuk, A., Stańczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C.,
Vincent, D., Michalski, M., Bousquet, O., al. Google research football: novel reinforcement learning environment. arXiv preprint arXiv: 1907. 11180, 2019.

---

[24] Lee, S., Sco, Y., Lee, K., Abbeel, P., and Shin, J. Offline-to-onling reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
[25] Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
[26] Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. Provably good batch reinforcement learning without great exploration. arXiv preprint (xXi:2007.08200 2020.
[27] Lowe, R.. Wu, Y. I., Tamar, A., Harb, J., Abbeel, O. P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-come compettiiv environments. In Advances in neural information processing systems (NeurlPS), 2017.
[28] Meng, L., Wen, M., Yang, Y., Le, Li, Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv:21 12.0284. 2021.
[29] Michie, D., Bain, M., and Hayes-Miches, J. Cognitive models from subcognitive skills. IEE control engineering series, 44:71-99, 1990.
[30] Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09339 2020.
[31] Nair, A.. Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
[32] Oliehoek, F. A., Amato, C., et al. A concise introduction to decentralized POMDPs. Springer, 2016.
[33] Pan, L.,, Rashid, T., Peng, B., Huang, I., and Whiteson, S. Regularized softmax deep multi-agent q-learning. Advances in Neural Information Processing Systems, 14:136511 2021.
[34] Pan, L., Huang, L., Ma, T., and Xu, H. Plan better amid conservatism: Offline multi-agent rein- forcement learning with actor rectification. In International Conference on Machine Learning. pp. 17221-17237. PMLR, 2022.
[35] Peng. B., Rashid, T., Schroeder de Witt, C., Kamienny, P.-A., Torr, P., Böhmer, W., and Whiteson, S. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34: 12112220-1222 2021.
[36] Peng. X. B., Kumar, A., Zhang, G., and Levine, S. ddvantage-weighhee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 1910.001777 2019.
[37] Rashid, T., Samvelyan, M., De Witr, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1485, 2018.
[38] Rashid, T., Farquhar, G., Peng, B., and Whiteson, S. Weighted qmix: Expanding monotonic value function factorisation. arXiv e-prints, pp. arXiv-2006, 2020.
[39] Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34, 2021.
[40] Samvelyan, M., Rashid, T., de Witt, C. C.S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.0440 1902.04043, 2019.
[41] Schulman, J., Wolski, F. Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707.06347, 2017.
[42] Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. International Conference on Machine Learning (ICML), 2019.
12

---

[43] Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V. F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, Z., Tuyls, K.. al. Valos-decomoositionn networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
[44] Tan, M. Mueti-agent reinforcement learning: Independent vs. cooperative agents. In Interna- tional Conference Machine Learning (ICML), 1993.
[45] Tseng, W.-C., Wang, T.-H., Lin, Y.-C., and Isola, P. Offline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Processing Systems, 2022.
[46] Wan, L., Liu, Z. Chen, X., Wang, H., and Lan, X. Greedy-based value representation for optimal coordination in multi-agent reinforcement learning. arXiv preprint arXiv:2112004454, 2021.
[47] Wang, J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008. 00088000622. 2020.
[48] Wu, J., Wu, H., Qiu, Z., Wang. J., and Long, M. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[49] Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arxiv: 1911199111111 1211..16.111. 1361, 2019.
[50] Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., and Goh, H. Uncer- tainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2005.08140. 2021.
[51] Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Dellman-consistee pessimism for offline reinforcement learning, Advances in neural information processing systems, 34, 2021.
[52] Yang. Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 34:10299-10311 2021.
[53] Yu, C., Velu, A., Vinitsky, E., Wang. Y., Bayen, A., and Wu, Y. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv: 10t::2/03.000 01955, 2021.
[54] Yu, T., Thomas, G. Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. Mopo: Model-basee offline policy optimization. arXiv preprint 2005:::9005.33399 2020.
[55] Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning, Advances in neural information processing systems, 34, 2021.
13

---

A Detailed Proof
A.1 Proof of Theorem 4.1
Proof. Similar to the proof of Theorem 3.2 in Kumar et al. [21], we first prove this theorem in the absence of sampling error, and then incorporate sampling error the end. By the derivation of the objective in Eq. to zero, we can compute the Q-function update induced in the exact, tabular setting(T* = n" and (a(s))) fa)()))))
Vs k a T
E
(A.1)
Then, the value of the policy, vk+1 can be proved to be underestimated, since:
Vik+ (arr(a ( T (s) allaat
" d
(A.2)
3
Next, we will show that (8) = Eat(( ssaa sa))ss)
4'la'la) 8y(a"
is always positive, when
En 1 (sa)s)
DCP (s) {alss
(A.3)
(a((a(s))
L (a'ls)
(A.4)
(a*)))
v'aalss)
En E=
Eg (7 To(('ls) (s)))
1
(A.5)
(aa)))
(aaaa(s)
(aaass
Eg (a r3((a\s)))
Eg n (a')s))
T
T (a
(A.6)
(a'(sss)s Ta ss)
-1= di
Σn
Eng
(a)))
+0
since Vi, E. (a'lss = Ege (a
(a'1s)
=
(A.7)
IV 0
(A.8)
As shown above, the D (s) 2 0, and ( = 0, iff r' = Ta(aa(s) (s). This implies that each value iterate incurs some underestimation, i.e. f & (s).
We can compute the fixed point of the recursion in Equation A.2 and get the following estimated policy value:
(s)=
(1 -
A 4 (a'))ss)
1
1 (s)
(A.9)
(a (8)
Because the (T - > is non negative and the (s) absence of sampling error, Theorem 4.1 gives lower bound.
2 0, it's easily to prove that in the
Incorporating sampling error. According the conclusion in Kumar al. write down the result with sampling error as follows:
21
directly
# (aaaass) (s)
I]
Cr.T, Rmax
V (x(() < V* (I-7PT) 1 15
T (als)
E
d
(s)+ (T - ppp
(a')s)
(1 /DD
(A.10)
14

---

So, the statement of Theorem 4.1 with sampling error is proved. Please refer to the Sec.D.3 in Kumar al. [2N For detailed proof. Besides, the choice of in this case prevent overestimation given by:
Cr.T.o-maz
(aaals) 5) N
I
2 max saaED (1 7)- sED
max
(al()
"g 23 ())
(A.11)
A.2 Proof of Theorem 4.2
Proof.
According to the definition, we can get the formulation of DCF DELL ()))) and DooL T (8) as follow:
A (a'ls)
(T (T, = (a
-1
(A.12)
n (a'1s)
t (a'ls) -120 0
A
(A.13)
(aals)
1=1
[(aas)
B(aas) -1)
(3) = Eanw(()ss)
(A.14)
(a'ls) (a'ls) (a
-120 0
(A.15)
Then, by taking the logarithm of Doqu (s),,, we get:
In In(Doqu(TT +1) Σ In (Eawa((le) <aa21s) (aaalss)
n
(A.16)
As 2AA= 1, it's obvious that
T (a)ss)) (
(3) 1 In In
Σ
where j = arg max EE
3r (A.17)
31 (a)ls)
a3
By combining equation A.16 and inequation A.17
we get
DTq ()( +1
(a'ls) (c(a)s))
DCF (T) +1
exp
Σ In
E Exi
(A.18)
im iml.iih
exp
(L(T()))) i=1,
Σ
where = arg max EE (A.19)
3
the second inequality is derived from the Jensen's inequality. As the Kullback- Divergence is non-negative, it's obvious that (TQ 2 DEQL(T, (s).. then can simplify the left-hand side of this inequality:
DooL(T (T, (s) CF CQL (T,, (s) 2 exp
Σ (L(T(())1 1=1,85
where j = arg max E
(A.20)
35
15

---

A.3 Proof of Equation 6
Proof. Similar to the proof of Lemma D.3.1 in CQL 21.. Q obtained by solving a recur- sive Bellman fixed point equation the empirical MDP M, with an altered reward, r(s, a)
| hence the optimal policy x² (als) obtained by optimizing the value under the CFCQL Q-function equivalently is characterized via Eq. 6.
(a'ls) Ex A 3'(a
A.4 Proof of Theorem 4.3
Proof. Similar Eq. 6, a a is equivalently obtained by solving:
TMa(a 8) + arg J [T,, (T, M) "1 - Band (e)))
(A.21)
Recall that Vs, n B, DCQL( () 2 0. We have
M( M) 2J (TM M) 2J( M) 1-
E --1 MA
Doqu B))
1
(s)
(A.22)
E
Bond (+)
(
(s)]
Then we give an upper bound of E greater than anywhere, have
Bondy (a)1 (DQQ((T* (s)) Due to the assumption that S is
M
T(ass) B(a 1] = Σ [[als]]
π (a)s)
(T, = Σ [aass|
1]
(a'ls)
(A.23)
<
(alss) - 1 m
-
c
Combining A.22 and Eq. A.23
get
1 1)
J (MMM 2 J(TT" M)
(A.24)
Recall the sampling error proved in 21 and referred to above (A.10). we can use bound the performance difference for any on true and empirical MDP by
Or.T,SRman
p(s)
[J(T,, - J(TT ((T,M))
Σ
(A.25)
(1 (1-
VID(S)]
then let sampling error := 2 Cr. Rmox Σ,
m(s) and incorporate it into A.24
(1-7)²
VID(o))
get
( M) J(TT", M)
sampling error
(A.26)
where sampling error constant dependent on the MDP itself and D. Note that during the proof do not take advantage of the nature of π²' Actually "" can be replaced by any policy The reason we use π that it can give that largest lower bound, resulting the best policy improvement guarantee. Similarly, DCF, can be bounded by -:
(a'ls) (aals) 1
DCF (T,
A:
n (aaa\s)) (a)\ss)(='(a((1))
(A.27)
R
Σ
d
1
16

---

A.5 Proof of Theorem 4.4
We first show the theorem of safe policy improvement guarantee for MACQL and CFCQL, separately. Then we compare these two gaps.
MACQL has safe policy improvement guarantee related to the number of agents
Theorem A.I. Given the discounted marginal state-distributin d's we define B( D)
Egwdy IVD (T, () (s) + 1]. The policy TiM (alss) is a ( -safe policy improvement over B in
the actual MDP M, i.e., A: 2 ((3 - SMA,, where CMA. 2 Crdd B M(TMA: DCQL) + ( 1) (J(mT M) - ()). PPs]
1-7 + R Or
(1-7)²
Proof. We can first get J MA -elattdd policy improvement guarantee following the proof of Theorem 3.6 in Kumar et al. 21:
MA M) 2J - (2
1 7 + RRmax
Cr.5
VIA viD(s)
M. DOQL)
(1 (1-
(A.28)
(M( M) - ((,
According to Eq. A.21 a obtained by optimizing J(T, M) with Door- regularizer. And Theorem 4.3 shows that DeQr. can be extremely large when the team size expands, which may severely change the optimization objective and affects the shape the optimization plane. Therefore,
A. M) may be extremely low, and keeping J(T A MM in Eq, A.28 results in a mediocre
policy improvement guarantee. To bound J M M), we introduce Eq. the following:
A.24 into Eq. A.28
we get
M 2B, - (2 Cr,s 1-7 71-
+ BBmax
VIA
B B(T DoQr)
/D((s)
(A.29)
7 1) (J(T",,)) - -1B,
This complete the proof.
We can get similar CCF satisfying J(T M) 2 J(3 for CFCQL, which is independent of n:
Cr.8 1-7 (1-
+ ERmaxCT.N VIAl
a 1 (--1) 1)-11(((== (B)
=2 =C
B(TCr, DC ) 1
(1 11-799 7)2
[
(A.30)
Then we can prove Theorem 4.4.
Proof. Subtract ÇCF from MA and we get:
Cr. MA CCF = 1-7" 71-
+ 7.maa
VIAL [D()
7 (_ --
(B(TiMA) DoQL)) B(TTrr
(A.31)
Let the right side 2 O, and we can get
logg max (1 ( a [D((s)
2 VIA
Ruman (TFF;DD0 B ( Deau!)
(Gr
(A.32)
According to Theorem 4.3.
1
(TFF; DCLL) E snd
=or IV
DCF DEPL(TTC ()()) + 1] < E
Ter
Iv
+ 1|
and
Ve (A.33)
17

---

In the meantime, we have
B (TCr, D E snud piea ly
DeQu(TMM: ()(s +122 sna rsea Iv (3. + +1] =
(A.34)
Therefore, relax the lower bound of to constant that
2 logg (
2 vIA [
(Cr,6 + 7.max Or,
('7 -1)
(A.35)
1-9
Ve
B Implement Details
B.1 Derivation of the Update Rule
To utilize the Eq. 4 for policy optimization, following the amalysis in the Section 3.2 in Kumar et al. [21]. we formally define optimization problems over each 4 (a') [s) by adding regularizer R(μ²). As shown below, we mark the modifications from the Eq. 4 in red.
min max
Σ
E (((,a)))
(B.36)
Br ( (0(ss.)) -
E 1=1
By choosing different regularizer, there are variety of instances within CQL family. As recom- mended Kumar et al. 121], we choose R(y") to be the KL-divergence against a Uniform distribution over action space, i.e. R(x²') - DKIlu" (n))( Then we can get the following objective for μ²::
max E Ex) (x) [f()]] H(,)) s.t. En'(x) - pf 2 0, Vr.
(B.37)
where Vs, f(x) = Q(, a "). The optimal solution is:
(x) exp(f(x)).
(B.38)
where Z is the normalization factor, i.e., 2 Σ, ex)(f(x)). Plugging this back into Eq. B.36 we get:
min a.E
Σ ArE -i: log Σ 1=1
exp(Q(s, a)) -a-ss(((ssa))
a²
(B.39)
1 B, Bs
- (
+ 2
B.2 Details for Computing X
To compute X, we need an explicit expression of n¹ and 3¹. In the setting of discrete action space, as use Q-learning, n be expressed by the Boltzman policy, i.e.
exp (EA i~3 :( a a(s,a),a
T' Taa)) Σn exp (Ea-iag- a a-i)
(B.40)
We use behaviour cloning to pre-train parameterized ((s) with three-level fully-connected network and MLE(Maximum Likelihood Estimation) loss.
With the explicit expression of 7' and 8i, can directly compute with Eq. 8 and Eq. While, in practice, we find the E., pT(s]
r'(s) may introduce extreme variance as its large scale and fluctuations,
18

---

which will hurt the performance. Instead, we take the logarithm of it and further reduced it to the Kullback-Leible Divergence as follow:
Vi, (,(s)) exp (-
(B.41)
For continuous action space, we use the deterministic policy like in MADDPG, whose policy distribution can be regared Dirac delta function. Therefore, approximate E (P()) following:
['(a) by the
(8)
(B.42)
(5))
(T)
Then we need to obtain an explicit expression of 3r. We first train a VAE 17 from the dataset to obtain the lower bound of 3². Let pa(a, 2|s) and (2(2|a, s) be the decoder and the encoder of the trained VAE, respectively. According to Wu et al. [8]] 8 (a) (s) can be explicitly estimated by (We omit the superscript for brevity):
pa |s)
log Ba 8) = log E (sla.s)
9p(2 a.
1
Pa (a, (1))
(B.43)
E (2(0ggs)aa.s)
log L
4 (2(1)
a,
def
log Tggrs(a 88; sp, o, L..
Therefore, we can sample from the VAE L times estimate 3 The sampling error reduces L increases.
C Experimental Details C.1 Tasks
Equal_ _iiee a multi multi-agent task which we design by simplify the space shape of Equal_Space to one-onneeet vre-dimension. There are n agents and they are randomly initialized to the interval [0, 2]. The state space is one- mie-dimensional bounded region in [o, m1.x(10,22 n)] and the local action space discrete, eleven-dimensional space, i.e. [0. 0000, 0.05, -0.1, -0.5, -1, 0.01, 0.05, 0.1, 0.5, 1], which represents the moving direction and distance each step. The reward is shared by the agents and formulated as 10 * (n 1) min_dis dinn_dii-lstt... _tppp_tttepp______ee stt___tep______ man dis which will spur the agents to cooperate to line_length spread out and keep the same distance between each other.
For Multi-agent Particle Environment and Multi-agent Mujoco, we adopt the open-source implemen- tations from Lowe et [27 and Peng et al. [35 respectively. And use the datasets and the adversary agents provided by Pan et al. 34).
For StarCraft II Micromanagement Benchmark,
the open-source implementation from Samvelyan et al. 40 and choose four maps with different difficulty and number of agents as the experimental scenarios, which is summarized in Table 4 We construct datasets with
QMIX (37 by collecting training or evaluating data.
C.2
StarCraft II datasets collection
The datasets are made based on the training process or trained model of QMIX(37). Specially, the preeraiid policy with medium
Medium or Expert datasets are sampled by executing performance level or fully-pretraincd policy. The Medium Replay datasets are exactly the replay buffer during training until the policy reaches the medium performance. The Mixed datasets are the equal mixture of M Medium and Expert datasets. All datasets contain five thousand trajectories, except for the Medium Replay.
19

---

Table 4: The details of tested maps in the StarCraft II micromanagement benchmark
Maps
Agents
Enemies
Difficulty
2s3z
Stalkers 3 Zealots
2 Stalkers & 3 Zealots
Easy
3s vs 5z
3 Stalkers
5 Zealots
Easy
5m vs 6m
Marines
6 Marines
Hard
6h ____s_8 8z
6 Hydralisks
8 Zealots
Super Hard
C.3 Baselines
BC: behavior cloning. In discrete action space, we train three-level MLP network with MLE loss. In continuous action space, we use the method of explicit estimation of behavior density in Wu et al. [48].
which is modified from VAE [17] estimator. TD3-BC(1T]: One of the SOTA single agent offline algorithm, simply adding the BC term to TD3 [2]. We use the open-source implementation and modify it CTDE version with centralised critic. [QL[18] and AWAC variants of advantage weighted behaviour cloning. We refer to the open-source implementation and implement CTDE version similar TD3-BC. MACQL:naive extension of conservative Q-learning, proposed in Sec. 3.3. We implement it based on the open-source implementation As the joint action space is enormous, we sample N actions for the logsumexp operation. MAICQ [52 :multi-ggen version of implicit constraint Q-learning by propose the decomposed multi-an agent
under implicit constraint. We use the open-source implementation" in discrete action space and cite the experimental
results in continuous action space from Pan et al. [34] OMAR [4] juses zeroth-order optimization for better coordination among agents' policies, based on independent CQL. (ICQL). We cite the experimental results in continuous action space from Pan et al. 334 and implement version in MADTKD [55 uses decision
discrete action space based on the open-source implementation transformer to represent each agent's policy and trains with knowledge distillation, As lack of open-source implementation, We implement it based on the open-source implementation of another Decision Transformer based method MADT
C.4 Resources
We use 2 servers run all the experiments. Each one has S*NVIDIA RTX 3090 GPUs, and AM 7H12 CPUs. Each setting is repeated for 5 seeds. For one seed in SC2, it takes about 1.5 hours. For MPE, minutes is enough. The experiments on MaMuJoCo cost the most, about hours for each seed.
C.5 Code, Hyper- prpper-prramters and Reproducibility
Please refer to this repository for the code, datasets and the hyper-parameten of method. For each dataset number 0, 1, 2, 3, 4, we use the seed 0, 1, 2, 4, respectively.
More results
D.1 Complete Results on MPE
Table shows the complete results of our methods and more baselines on Multi-agent Particle Environment. Some results are cited from Pan et al. [34].
httpe///gthhbb.oo////aaaaa/////t
Biiiiine--------e trained-MM MrMMMMulti--eee Decision Transformer
20

---

Table 5: Complete results Multi-agent Particle Environment.
Env
Dataset
MAICQ
MATD3-BC
ICQL
OMAR
MACQL
IQL
AWAC
CFCQL
Random
6.3+3.5
9,8+4,9
24.0=9.8
34.4+5.3
45.6+8.7
5.5+1.1
0,5+3.7
62.248.1
Medium-replay
13.645.7
15.455.6
20.0=8.4
379==1.3
25.5+5.9
10.844.5
2.743
52.249.6
CN
Medium
29.3+5.5
29.314.8
34.1=7.
47.921899
14.3+20.2
28.243.9
25.744.1
65.0=10.2
Expert
104.0=3.4
108.3+3.3
98.2+5.2
114.922.6
12.2431
103.7+2.5
103.3+3.5
112+4
Random
2.242.66
5.7=3.5
5.048.2
11.1+2.8
25.2+11.5
1.3+1.6
0.2=1.0
78.5
Medium-replay
34.5+27.8
28.7=20.9
24.8417.3
47.1=15.3
11.9992
23.2+12
8.325.3
71.126
PP
Medium 63.3+20.0
15..5
61.7423.1
66.7=23.2
55+43.2
53.6#19.9
50.9=19.0
68.5+21.8
Expert
113.0#14.4
115.2+12.5
93.9*14.0
116.2+19.8
108.4+21.5
10..3+10.1
106,5+10.1
118.2+13.1
Random
1.04:3.2
2.8+5.5
0.642.0
5.9=5.2
11.7+11
2,9+4.0
-2.4+2.0
68+20.8
Medium-replay
12.0:9.1
17.488.1
29.6
42.9=19.5
13,2+16.2
41.5+9.5
8.9=5.1
73,4+23,2
World
Medium
71.9+220.0
73.4+9.3
58.6*11.2
74.6=11.5
67.4a48.4
70.5=15.3
63.9=14.2
93.83318
Expert
109.5+22.8
110.3421.3
71.9+28.1
110.4+25.7
99.7431
107 8+17.7
107.6+15.6
119.7426.4
Random
Med-Rep
Medium
Expert
120
220-
Score
635
000
Figure Ablations on World.
D.2 Temperature Coefficient in Continuous Action Space
We carry ablations of on MPE's map World in Fig. We find that although. the best differs in different datasets, the overall performance is not sensitive which verifies the theoretical analysis that any simplex of À that Σm=1 A can induce an underestimated value function.
D.3 Ablation CQL
We carry out ablations of a MPE's map World in Fig. We find that plays more important role for team performance on narrow distributions (e.g., Expert and Medium) than that on wide distributions (e.g., Random and M edium Replay).
D.4 Component Analysis on Counterfactual style
In the environment MaMuJo, except for the counterfactual Q function, we also analyze whether the conuterfactual treatment in CFCQL can be incorporated in other components and help further improvement in Table
We find that the counterfactual policy improvement is critical for this
Random
Med-Rep
Medium
Expert
302
540
Spore
Figure 5: Ablations of on World.
21

---

Table 6:
Component Analysis
on MaMuJoCo.
CF_T: computing target Q by CF the policy improvement (PI) by Eq. 10, other-
Enj-Uni([11, Eg iDD a (a(s, a).
wise using MADDPG's PI.
Dataset
Default
+CF_T
-CF_P
MACQL
Random
39.7+4.0
48.7+1.8
23,949.2
5.3+0.5
Med-Rep
59.5#8.2
58,949.6
43.545.6
36.7=7.1
Medium
80.5+9.6
76.2+12.1
43.8+7.8
51.5+26.7
Expert
118.5#4.9
118.1
3.7+3.1
50.1+20.1
environment. With CF_ the method shows great performance gain on narrow data distribution, e.g., the Experl dataset.
50
500
5000
50000
0.8
Test Winning rate
0.2
Medium
Mixed
Expert
Size the DataSet
Figure 6: Hyperparameters examination on the size of datasets.
D.5 Analysis on Size of Dataset
An additional study examines dataset size effects. We generate full dataset with 50, 000 trajectories for each type in map 6h_us 2s__2, creating smaller datasets by sampling 5, 50, 500, 5000 trajectories. 6 displays CFCQL's testing winning rate for varying dataset sizes. It's notable that ensure fairness, the maximum number of training steps for all datasets and algorithms on the SMAC environment is fixed 1e7. In this additional study, however, trained the CFCQL algorithm until convergence to eliminate the impact of underfitting. The results demonstrate that larger datasets contribute to improved convergence performances, thus confirming the scalability of CFCQL for larger data samples.
E Discussion
E.1
Overestimation Offline RL
We offer an intuitive explanation for the phenomenon overestimation caused by distribution shift in this section. For a rigorous proof, we refer readers to the related works.
In offline RI,, key challenge arises due to the distribution mismatch between the behavior policy-the the policy responsible for generating the data-and the target policy, which is the policy one aims to improve. This mismatch can result in extrapolation errors during value function estimation, often leading to overestimation. Specifically, during the policy evaluation stage, the dataset may not encompass possible state-action pairs in the Markov Decision Process (MDP), leading inaccurate Q function estimates for unseen state-action pairs. These estimates may be either too high or too low compared to the actual Q values. Subsequently, in the policy improvement stage, the algorithm tends to shift towards actions that appear to offer higher rewards based on these overestimated values. Unlike online RL, which can implement the current training policy in the environment to obtain real feedback and thereby correct the policy's direction, offline RI lacks this corrective mechanism unless careful loss design employed. When using common bootstrapping methods like temporal difference learning for training the Q function, these overestimation errors propagate, affecting estimates
22

---

for other state-action pairs. This can result in a chain reaction of overestimations, potentially causing an exponential explosion of the Q function.
23


Reviews:
[{"review_id": "Ak6XJWd1Zg", "reviewer": "Reviewer_rt2G", "summary": "\t\t \t \t \t\t\nThis paper addresses challenges in Offline Multi-Agent Reinforcement Learning (MARL), which suffers from severe distribution shift issues and high dimensionality. To overcome these problems, the authors propose a novel MARL algorithm, CounterFactual Conservative Q-Learning (CFCQL). Unlike conventional methods that treat all agents as a single high-dimensional entity, CFCQL separately computes conservative regularization for each agent in a counterfactual manner and linearly combines them for an overall conservative value estimation. The authors demonstrate that CFCQL maintains the underestimation property and performance guarantee similar to single-agent methods but offers improved regularization and safe policy improvement bounds that are independent of the agent number. This method is thus advantageous, especially with large agent counts. Experimental validation on various environments and datasets shows that CFCQL outperforms existing methods significantly.\t\n\n", "questions": "See weakness above.\n\n", "limitations": "* The presentation needs to be improved in terms of clarity, and rigor.\n\n* The method is still employing centralized training, which limits its scalability.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The structure of the presentation is generally clear.\n* The considered problem is interesting and of importance.\n\n\n\n", "weaknesses": "* The motivation for the algorithm is unclear, e.g. why considering counterfactual is helpful for offline MARL?\n* The theoretical analysis is not well elaborated in context. E.g. the implications of theorems can be better elaborated.\n* Some terms are not clearly defined, e.g. “PI”, “e_i” in Eq.(8). \n* How algorithm 1  works with continuous or discrete action space is not clear. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback!\n\n●**Q1: The motivation for the algorithm is unclear, e.g., why considering counterfactual is helpful for offline MARL?**\n\nWe apologize for any confusion resulting from the unclear statement in our paper. \n\nThe motivation for this study is rooted in the observation that directly transferring CQL into Multi-Agent settings, wherein the joint action is penalized, leads to an exponential increase in pessimism as the number of agents increases. Consequently, this results in an overly pessimistic value function. \n\nTo mitigate this issue, it is a potential solution to separately penalize each agent's out-of-distribution (OOD) actions, then the penalty term will grow linearly instead of exponentially. This idea aligns perfectly with the counterfactual approach in MARL[1]. Specifically, counterfactual approaches aim to obtain each agent's counterfactual baseline by marginalizing out the agent's action while keeping the other agents' actions fixed to address the challenges associated with multi-agent credit assignment.\n\nIn the context of our study, we successfully adapted the counterfactual method in offline multi-agent settings. This is achieved through the separate penalization of each agent's OOD actions, while holding the other agents' actions still sampled from datasets. Theoretically, this adaptation holds promise in alleviating the out-of-distribution problem with reduced pessimism. Moreover, the empirical results from our experiments serve to further validate the effectiveness of the counterfactual method in the domain of Offline Multi-Agent RL.\n\n●**Q2: The theoretical analysis is not well elaborated in context, e.g., the implications of theorems can be better elaborated.**\n\nIn this paper, we aim to theoretically analyze the pessimism of CFCQL and highlight its advantages over MACQL. In Theory 4.1, we rigorously demonstrate that our proposed updating rule, as represented by Eq. 4, leads to a lower bound of the value function. This finding validates the pessimistic property of CFCQL, which is necessary for offline settings. Moving forward, in Theory 4.2, we delve into a comparative study of the scale differences between CFCQL and MACQL in terms of their degree of pessimism. Our analysis reveals that the degree of pessimism exhibited by MACQL surpasses that of CFCQL and that the scale difference between the two methods tends to increase with the number of agents. Building upon the findings from Theory 4.2, we engage in a comprehensive discussion regarding the performance bound and safe policy improvement guarantee of both methods in Theory 4.3 and Theory 4.4, which allows us to showcase the dominance of CFCQL in comparison to MACQL, highlighting the superiority of our proposed approach.\n\n●**Q3: Some terms are not clearly defined, e.g., “PI”, “e_i” in Eq. (8).**\n\nWe sincerely apologize for the lack of clarity in our paper and any resulting confusion it may have caused. Allow us to clarify the notation in question. In line 281, \"PI\" stands for \"policy improvement,\" denoting the iterative progress of policy update in the context of Reinforcement Learning. Additionally, in Eq.8, the symbol “e_i” represents a one-hot vector, wherein the i-th element assumes a value of 1 while all other elements remain 0.\n\n●**Q4: How algorithm 1 works with continuous or discrete action space is not clear.**\n\nWe sincerely apologize for any confusion that may have arisen from our algorithm implementation. Allow us to clarify the algorithm implementation for both discrete and continuous action spaces in our work.\n\nFor the discrete action space, we utilized the QMIX algorithm, a well-known Multi-Agent Reinforcement Learning (MARL) Q-Learning method specifically designed for discrete action spaces as our backbone algorithm. Upon QMIX, we implement our counterfactual pessimism by the updating rule in Eq. 7. \n\nFor the continuous action space, we opted for the MADDPG algorithm, a MARL algorithm based on deterministic policy gradient, as our backbone algorithm. In this setting, in addition to the replacement of update rule for the Q function (as denoted by Eq. 7), we also adapted the update rules for the policy function $\\pi$ (as represented by Eq. 10) to align with the nature of the continuous action space scenario.\n\nWe hope that these clarifications suffice to remove any ambiguity surrounding our algorithm implementation. \n\nReference:\n\n[1] Foerster J, Farquhar G, Afouras T, et al. Counterfactual multi-agent policy gradients[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1).\n"}, {"review_id": "5vkTkbbE0l", "reviewer": "Reviewer_PGy4", "summary": "This paper proposes a novel offline multi-agent reinforcement learning algorithm called Counterfactual Conservative Q-Learning (CFCQL) to address the overestimation issue and achieve team coordination at the same time. The algorithm calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. The paper compares CFCQL and MACQL theoretically and shows that CFCQL is advantageous to MACQL on the performance bounds and safe policy improvement guarantee as the agent number is large. The paper conducts experiments on commonly used multi-agent environments to demonstrate that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.", "questions": "- How to get $a_i$ in eq.7? Or what's the range of the summation?\n- To my understanding, does CFCQL update $Q$ for each agent's action iteratively instead of jointly in MACQL? In this case, does the order of the updates matter? And why sample only OOD actions from one agent instead of two or more if two agents' action are highly related?", "limitations": "The paper does not explicitly mention any limitations of the proposed method. The paper has no potential negative societal impact.\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Theoretical comparison of the proposed algorithm with existing methods to show its advantages in terms of performance bounds and safe policy improvement guarantee.\n- The paper provides a detailed explanation of the proposed algorithm and its counterfactual approach to conservative value estimation.\n- Conducting experiments on multiple environments to demonstrate the effectiveness of the proposed algorithm.", "weaknesses": "- The paper seems to be a simple combination of CQL and MARL.\n- Although the paper shows that the proposed algorithm is theoretically superior to existing methods when the agent number is large, it does not provide empirical evidence to support this claim.\n\ntypos\n- Section \"conclusion\" -> \"Conclusion\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback!\n\n●**Q1: No empirical evidence to support the superior of our method on large agent number $n$.**\n\nWe apologize for any unclear experimental instructions in our paper. The evidence supporting our claim can be found in Section 5.1, specifically in the demo labeled 'Equal Line'. In Figure 2, it is evident that the performance of MACQL degrades significantly as the number of agents increases, while the performance of CFCQL remains relatively stable. These results strongly support the conclusion we presented in Section 4.3, highlighting that CFCQL provides better policy improvement guarantees compared to MACQL when the number of agents $n$ is sufficiently large. \n\n●**Q2: The paper seems to be a simple combination of CQL and MARL.**\n\nWhile CFCQL shares some similarities in loss form with CQL, it differs fundamentally from a simple combination of CQL and MARL (MACQL). Our theoretical analysis demonstrates that MACQL leads to an exponential increase in pessimism as the number of agents increases, resulting in an overly pessimistic value function. To address this issue, a potential solution is to separately penalize each agent's out-of-distribution (OOD) actions, allowing the penalty term to grow linearly rather than exponentially.\nCFCQL successfully adapts the counterfactual method to offline multi-agent settings by independently penalizing each agent's OOD actions while keeping the other agents' actions sampled from datasets. This adaptation holds promise in alleviating the out-of-distribution problem with reduced pessimism. Moreover, the empirical results from our experiments further validate the superiority of CFCQL over MACQL. These unique contributions distinguish CFCQL and demonstrate its effectiveness in multi-agent scenarios.\n\n●**Q3: How to get $a_i$ in eq.7? Or what's the range of the summation?**\n\nWe sincerely apologize for the lack of clarity in our paper and any resulting confusion it may have caused.  The summation range of $a_i$ in Eq. 7 corresponds to the action space of agent $i$, which is similar to the concept in CQL. In terms of implementation details, for the discrete action space, we perform the summation directly over the action space using the standard torch.logsumexp() function, which allows us to efficiently compute the required summation. However, for the continuous action space, we employ an importance sampling method inspired by the CQL. Specifically, we generate action samples from both a uniform-at-random distribution (Unif(a)) and the current policy. These action samples are then utilized in conjunction with importance sampling to compute the summation.\n\n●**Q4: Does CFCQL update $Q$ for each agent's action iteratively instead of jointly in MACQL? In this case, does the order of the updates matter?**\n\nIn CFCQL, the symbol $Q$ refers to the joint state-action function shared by all $n$ agents, which undergoes simultaneous updates involving both the counterfactual terms and the TD error. And we train the agents simultaneously by taking the summation of all $n$ counterfactual terms as the training loss.\n\n●**Q5: Why sample only OOD actions from one agent instead of two or more if two agents' action are highly related?**\n\nThe significance of the number of agent with OOD actions directly influences the disparity between the estimated value functions and the true value functions.  It is possible to sample OOD joint-actions for two agents as MACQL does. However, as illustrated in theorems 4.1 and 4.2 in our paper, this leads to an increased order of pessimism, which is squared compared to CFCQL.  And as more agents are involved, the induced pessimism will explode exponentially. However, as Theorem 4.3 demonstrates, a greater gap between the estimated and true value functions will lead to a larger discrepancy between the performance of the learning policy and the optimal policy, which implies that the huge gap brought in by MACQL will severely affect the performance of the learned policy. Therefore, sampling OOD joint-actions from two or more agents may perform much worse than sampling just single OOD action from one agent. That is our reason behind the choice of sampling only OOD actions from one agent.        \n"}, {"review_id": "aFvYcYqw4z", "reviewer": "Reviewer_dhyd", "summary": "Offline Reinforcement Learning in the multi-agent setting suffers from the combined effects of distribution shift and increasing number of agents. An exponential blowup of the action space in addition to Out-Of Distribution (OOD) actions hinders performance of RL agents. The work tackles these phenomena by proposing CounterFactual Conservative Q Learning (CFCQL), an algorithm that learns conservatively from static offline datasets. CFCQL differs from the naive multi-agent CQL approach as it conservatively trains each agent separately using the pessimistic regularization. Regularization terms are then linearly combined for global value estimation. This prevents agents from being excessively conservative while maintaining theoretical guarantees of underestimation and safe policy improvement as in CQL. In practice, weighted contributions of conservative penalties are realized either using a one-hot encoding or softmax with temperature scaling. In the case of continuous actions, counterfactual Q function updates of the gradient are utilized. Experiments are carried out on a range of discrete and continuous multi-agent benchmarks with promising improvements.", "questions": "* What is the formal problem definition of offline RL? What is a behavior policy? What is distribution shift?\n* What is the interpretation of Figure 3 (b)? How does an increase in the size of dataset lead to a decrease in agent performance? How does CFCQL scale for larger data samples?\n* Can you please explain the reasoning behind off-policy/online RL methods being relevant baselines? Can CFCQL be compared to IQL, TD3-BC or AWR even if on small toy tasks?\n* How does CFCQL address distribution shift? How does CFCQL generalize to new task? Can CFCQL be adapted to new multi-agent tasks or transferred to different kinds of agents following pretraining?", "limitations": "Authors have discussed limitations and future work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* The paper presents an amenable combination of design choices.\n* Empirical results are promising and extensive.", "weaknesses": "* **Writing and Presentation**: My main concern is the writing and presentation of the paper. The paper is not well written and presented in an unorganized manner. Technical claims made by authors are vague and informal. Explanations are not well supported by intuition or insights and the frequency of grammatical errors is too high. Specifically, sections 1 and 2 motivate the work with high-level and vague statements. Section 3.3 does not formally explain the offline RL problem (of maximizing the expected discounted return), behavior policy or distribution shift. Section 4 builds the algorithm using informal vocabulary. Section 4.4 presents theoretical guarantees without any intuition or interpretation and section 6 summarizes the paper informally. In my view, the paper's presentation requires significant attention.\n* **Dataset Ablations**: I am having trouble understanding results on dataset ablations presented in figure 3 (b). Ideally, as the number of data samples grow, the performance of agents improves. With larger static datasets agents have access to more in-distribution samples and a broader coverage of the underlying MDP. However, figure 3 (b) demonstrates that performance of agents drops for 50,000 samples. It would be helpful if the authors can explain this result or its interpretation. In its current form, it appears that the approach may not scale well to larger multi-agent learning datasets for real-world applications.\n* **Baselines**: While the paper includes relevant multi-agent learning baselines, it is worth noting that none of these baselines were designed for offline RL. All methods were developed as off-policy or pure online RL methods that aggregate new experience. With that said, the only baseline of interest is MACQL since it leverages the CQL penalty designed for static datasets. Authors should consider offline-RL baselines that may be adapted for multi-agent learning. For instance, IQL[1] and TD3-BC[2] are state-of-the-art offline RL methods which might be useful for comparison. Similarly, AWR[3] is another method that imitates dataset interactions.\n* **Transfer and Finetuning**: The paper claims that CFCQL addresses distribution shift and generalizes better. However, this claim is not well supported. Ideally, robustness and generalization ability of an algorithm are tested by transferring it to new unseen scenarios. Authors should consider finetuning agents or adapting them to a new task even if on a small toy example. This will help verify the claims of CFCQL addressing distribution shift.\n\n\nWhile experiments and results are promising, overall presentation and writing of the paper needs significant improvement.\n\n## Minors\n\n* line 17: man-made -> synthetic\n* line 23: ~from~\n* line 24: ~the~\n* line 24: highly -> high\n* line 37: agent number -> number of agents\n* line 44: ~just~\n* line 55: bounded from below -> lower bounded\n* line 63: too much -> excessive\n* line 66: contributes CQL -> contributes to CQL\n* line 69: sampled in the dataset -> sampled in-distribution\n* line 70: agents number -> number of agents\n* line 72: man-made -> synthetic\n* line 79: advantagous -> advantageous\n* line 149: dataset distribution -> behavior policy\n* Figure 1: stays -> stay\n* line 171: even worse -> more significantly\n* line 175: need learn -> need to learn\n* line 178: exponentially -> exponential\n* line 183: style -> update\n* line 198: contributes regularization -> contributes its regularization\n* line 201: in the dataset -> in-distribution\n* line 238: performances -> performance\n* line 257: contributes penalty -> contributes to penalty\n* line 258: style -> encoding\n* line 307: agents number -> number of agents\n* line 314: basically -> mostly\n* Figure 3: temporature -> temperature\n* line 372: conlcusion -> Conclusion\n* line 373: lack of -> lacks\n* line 374: from theories -> using theory\n* line 375: exponentially -> exponential\n* line 379: exponentially -> exponential\n* line 381: achieve -> achieves\n* line 381: Some -> Ablation studies\n* line 382: also made -> conducted\n\n[1]. Kostrikov et. al., Offline Reinforcement Learning with Implicit Q-Learning, ICLR 2022.  \n[2]. Scott Fujimoto, Shixiang Shane Gu, A Minimalist Approach to Offline Reinforcement Learning, NeurIPS 2021.  \n[3]. Peng et. al., Advantage Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, arXiv 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and detailed syntax error check!\n\n●**Q1: Presentation problem. What is the formal problem definition of offline RL? What is a behavior policy? What is distribution shift?**\n\nWe apologize for the unclear statement, and we are committed to improving the presentation. To achieve this, we will seek assistance from native speakers. \n\nAs stated in Section 3.1, the online MARL problem concerned in this work is formulated in the context Partially Observable Markov Decision Process (POMDP). And the target is to maximize the cumulative reward $E_{\\pi}[\\sum_{t=0}^{\\infty}\\gamma^t r(s_t, a_t)]$. When transferred to offline MARL, the primary distinction is that the training data is a static dataset of transitions, denoted by $D={(s_t,o^i_t, a^i_t,s_{t+1},o^i_{t+1}, r^i_t)}$, where $i\\le n$ means the agent index, and $t$ denotes the time-step. Additionally, the agent no longer has the ability to collect data by interacting with the environment. \n \nThe behavior policy $\\beta$ denotes the distribution over states and actions in D and the transitions in D could be interpreted as being sampled according to the behavior policy, i.e., $s\\sim d^\\beta(s)$, $a\\sim \\beta(a|s)$, where $d^\\beta(s)$ means the state visitation distribution under policy $\\beta$.\n\nOffline RL algorithms encounter the distribution shift issue. The distribution of the behavior policy generally differs from the distribution of the learned policy. Consequently, when updating the Q-values with Bellman backups, the actions sampled from the learned policy may be out-of-distribution (OOD) actions, potentially leading to the overestimation of Q-values[1]. Due to the inability to interact with the environment to correct such errors, the distribution shift issue becomes a major challenge in offline RL.\n\n●**Q2: What is the interpretation of Figure 3 (b)? How does an increase in the size of dataset lead to a decrease in agent performance? How does CFCQL scale for larger data samples?**\n\nThe decrease in the performance of larger datasets (50k trajectories) in Figure 3 (b) occurs because of the under-fitting issue. To ensure fairness, the maximum number of training steps for all datasets and algorithms on the SMAC environment is fixed at $1\\times 10^7$. However, this may result in termination of trainings before convergence is achieved, resulting in the under-fitting issue. \n\nTo verify our claim aforementioned, we retrained the CFCQL on datasets containing 5,000 or 50,000 trajectories until convergence. The performances of the converged agents and the steps taken to achieve convergence are presented in the table below. The results demonstrate that larger datasets contribute to improved convergence performances, thus confirming the scalability of CFCQL for larger data samples.\n|Performance|Medium|Expert|Mixed|\n|-----------|--------|--------|---------|\n|5k(showed in the paper)|0.41±0.04|0.7±0.06|0.49±0.08|\n|50k(showed in the paper)|0.36±0.13|0.63±0.09|0.5±0.13|\n|5k(converged)|0.43±0.07|0.75±0.05|0.65±0.08|\n|50k(converged)|**0.47±0.07**|**0.79±0.07**|**0.67±0.07**|\n\n|Convergence Steps($\\times 10^7$)|Medium|Expert|Mixed|\n|-----------|--------|--------|---------|\n|5k|2|2|2|\n|50k|2|2|5|\n\nRegarding the Medium-Replay dataset, we have discovered a bug in our code and the maximum number of trajectories in the Medium-Replay datasets is 5000 instead of 50000. Since only 5000 trajectories are generated when agents are trained to medium performance, we cannot fill a replay buffer with 50000 trajectories for this kind of dataset. Therefore, the training results pertaining to Medium-Replay datasets with 5,0000 trajectories are invalid. We deeply apologize for this mistake and have rectified it in our code. We will revise our paper accordingly.\n\n●**Q3: How does CFCQL address distribution shift? How does CFCQL generalize to new task?**\n\nWe acknowledge that the distribution shift problem in offline RL is distinct from the generalization problem. In offline RL, the distribution shift refers to the divergence between the trajectory distribution generated by the behavior policy $\\beta$ (i.e., the dataset) and the imagined distribution of trajectories generated by the current training policy $\\pi$. The main challenge in offline RL is to provide accurate feedback to the current policy $\\pi$ using the data distribution induced by another policy $\\beta$, which constitutes the distribution shift problem. We tackle this problem with a counterfactual regularizer to penalize the action that is not sampled from the dataset, i.e., the left part of Eq. 4 in the original paper.\n\nIn our work, both the dataset and our trained policy share the same task and reward function, and we do not require agents to possess generalization abilities for transferring to a new task. Instead, we focus on addressing the distribution shift problem to effectively learn from offline data and improve policy performance within the given task.\n\n●**Q4: Can you please explain the reasoning behind off-policy/online RL methods being relevant baselines? Can CFCQL be compared to IQL, TD3-BC or AWR even if on small toy tasks?**\n\nDue to the page limit, please refer to the **public rebuttal block** with name \"Author Rebuttal by Authors\" on the top of this page for the explanation of lacking baselines.\n\nReference:\n\n[1] Fujimoto, Scott, David Meger, and Doina Precup. \"Off-policy deep reinforcement learning without exploration.\" International conference on machine learning. PMLR, 2019."}, {"review_id": "O9XGDB8DI1", "reviewer": "Reviewer_EZbv", "summary": "The paper addresses the problem of offline multi-agent reinforcement learning. A way to make conservative updates to the Q-function is proposed, extending, non-trivially, CQL to multi-agent settings. Theory and experiments clearly validate the approach.", "questions": "- In Section 4.1, what are the optimal Q-values of the Toy MDP ?\n- What is $\\beta$ ? It appears first in Section 3.3 and I don't see it defined, even though it seems quite important in the sequel (for instance, it appears twice in the proposed algorithm).\n- Why is the method called counterfactual ? I suggest adding an intuitive explanation to the document.\n- What is the performance of single agent independent Q-learning (actually DQN) and of independent CQL (actually DQN) on the SMAC environment ? I believe it would be quite important to make clear on the table of the results that coordination is really necessary to perform the task and that ICQL is not enough.", "limitations": "I can not see the limitations of the work nor I foresee a negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The method is both well motivated as well as theoretically and empirically validated. The method is experimentally shown to be substantially better than all the state-of-the-art baselines considered.", "weaknesses": "The introduction can be improved in terms of written language. Moreover, I would like to see a couple of straightforward independent learning baselines added to the StarCraft results. See my questions below.\n\nMinor comments:\n- I do not understand the last sentence of 3.3;\n- Some sentences to improve: figure 1(a), first sentence; first paragraph of the introduction;\n- I suggest giving an intuitive meaning to Theorems 1 and 2 after they appear, as well as an intuitive meaning of the whole theory of Section 4.3 in the end of the Section.\n- typos: \"temporature\" in the caption of Figure 3(a); capitalize \"conclusion\" in the title of Section 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comment!\n\n●**Q1: Typos, sentence improvement, and intuition explanation.**\n\nAs per your suggestion, we have fixed the typos and reorganized the sentences. The intuition behind our theories is as follows: Theorems 4.1 and 4.2 in our paper illustrate the differences in the extent of underestimation of value functions between CFCQL and MACQL. When MACQL samples $n$ OOD actions, it leads to significantly larger gaps between estimated V and true V compared to CFCQL. Additionally, Theorem 4.4 demonstrates that a greater gap between value functions leads to a larger discrepancy between the performance of the trained policy and the behavior policy. These updates and explanations will be reflected in the revised script.\n\n●**Q2: The explanation of the last sentence of 3.3.**\n\nWe apologize for the unclear statement. We want to convey that with a large enough hyper-parameter $\\alpha$ in Eq. 2 as the training loss, we can obtain a  $Q$ function, and the corresponding value function, which is lower than the real value function $V^{\\pi}(s)=E_{\\pi}[\\sum_{i=0}^{\\infty}\\gamma^i r_i]$ on the same MDP at any state. Based on this, the overestimation problem in single agent RL can be resolved.\n\n●**Q3: In Section 4.1, what are the optimal Q-values of the Toy MDP?**\n\nThe Multi-Agent Markov Decision Process (MMDP) proposed in Section 4.1 is an infinite Markov Decision Process (MDP) task, wherein the maximum reward attainable at each step is 1, contingent upon all agents reaching and remaining in state S2. The discount factor (\\gamma) was set to 0.8, and consequently, the optimal Q-value is determined to be 1/(1-0.8)=5, as illustrated in Fig. 1(b) with a dashed line.\n\n●**Q4: What is $\\beta$? It appears first in Section 3.3 and I don't see it defined, even though it seems quite important in the sequel (for instance, it appears twice in the proposed algorithm).**\n\nWe sincerely apologize for the unclear definition in our paper and any resulting confusion it may have caused.  In our work, we denote $\\beta$ as the behavior policy responsible for generating the dataset D. It is important to note that $\\beta$ is often represented as an implicit distribution, and in our proposed algorithm, we resort to employing an empirical policy approximation through the behavior-cloning method.\n\n●**Q5: Why the method is called counterfactual? I suggest adding an intuitive explanation to the document.**\n\nWe refer to our method as 'counterfactual' because its structure bears resemblance to counterfactual methods commonly used in multi-agent reinforcement learning (MARL). This involves obtaining each agent's counterfactual baseline by marginalizing out a single agent's action while keeping the other agents' actions fixed. The intuitive rationale behind employing a counterfactual-like approach is that by individually penalizing each agent's out-of-distribution (OOD) actions while holding the other agents' actions constant from the datasets, we can effectively mitigate the out-of-distribution problem in offline MARL with reduced pessimism. As per your suggestion, we will add the explanation to the revised version.\n\n●**Q6: What is the performance of single agent independent Q-learning (actually DQN) and of independent CQL on the SMAC environment?**\n\nWe conducted additional experiments to demonstrate the performance of independent Q-learning (IDQN) and independent CQL (ICQL) on the SMAC environment as listed below. As expected, IDQN proved to be inadequate in handling the offline tasks due to the absence of a specialized mechanism to address the distribution shift issue. ICQL demonstrated promising performance, primarily attributed to its pessimistic design, yet it fell short in achieving high-level performance due to the limitations in coordination imposed by the independent learning paradigm. The supplementary experiments underscore the essentiality of integrating pessimism and CTDE paradigm within the offline MARL domain.\n\n| Map | Dataset | CFCQL| IDQN | ICQL |\n| ------- | --------- | ------- | --------- | ---------------- |\n|5m_vs_6m|Medium| **0.29±0.05** |0.00±0.00| 0.21±0.04|\n||Medium_Replay|**0.22±0.06**|0.00±0.00 | 0.21±0.03|\n||Expert|**0.84±0.03**|0.00±0.00| 0.73±0.06|\n||Mixed|**0.76±0.07** |0.00±0.00| 0.72±0.07| \n|6h_vs_8z|Medium| **0.41±0.04**|0.00±0.00| 0.35±0.04|\n||Medium_Replay|**0.21±0.05**|0.00±0.00 | 0.09±0.05|\n||Expert|**0.7±0.06**|0.00±0.00| 0.49±0.10|\n||Mixed|**0.49±0.08** |0.00±0.00|0.33±0.06 | \n\n●**Q7: Lack of discussions on limitations and broader impacts:**\n\nDue to the page limitation, we have put these to Appendix E.\n\n"}]

Author rebuttal:
●Q1: Explanation on current baselines and more baseline results.

We acknowledge that the baselines used in our paper (OMAR, ICQ, and MADTKD) are designed for **offline** MARL, not off-policy MARL. We recognize that our paper lacks sufficient comparison with single agent offline RL methods except TD3-BC, the results of which have been listed in the original paper, Table 3 and Appendix Table 2. 

Furthermore, following the request of Reviewer dhyd, we conducted additional experiments and have now incorporated the results of IQL and AWR into our comparison on **all** datasets appeared in the paper. To ensure better agent cooperation in continuous action spaces, similar to CFCQL, we applied the CTDE paradigm to both IQL and AWR. This involved utilizing a centralized Q/Value function while maintaining decentralized policies. Additionally, to enhance the stability of AWR, we adopted the more robust actor-critic alternate, AWAC[1].

The results on MaMuJoCo are listed below:

|Datasets|CFCQL|TD3-BC|IQL|AWAC|
|----------|--------|---------|-----|-------|
|Random|**39.7±4.0**|7.4±0.0|7.4±0.0|7.3±0.0|
|Med-Rep|**59.5±8.2**|27.1±5.5|58.8±6.8|30.9±1.6|
|Medium|80.5±9.6|75.5±3.7|**81.3±3.7**|71.2±4.2|
|Expert|**118.5±4.9**|114.4±3.8|115.6±4.2|113.3±4.1|

The results on MPE are listed below:

|Maps|Datasets|CFCQL|TD3-BC|IQL|AWAC|
|------|----------|--------|---------|-----|-------|
|CN|Random|**62.2±8.1**|9.8±4.9|5.5±1.1|0.5±3.7|
||Med-Rep|**52.2±9.6**|15.4±5.6|10.8±4.5|2.7±3|
||Medium|**65.0±10.2**|29.3±4.8|28.2±3.9|25.7±4.1|
||Expert|**112±4**|108.3±3.3|103.7±2.5|103.3±3.5|
|PP|Random|**78.5±15.6**|5.7±3.5|1.3±1.6|0.2±1.0|
||Med-Rep|**71.1±6**|28.7±20.9|23.2±12|8.3±5.3|
||Medium|**68.5±21.8**|65.1±29.5|53.6±19.9|50.9±19.0|
||Expert|**118.2±13.1**|115.2±12.5|109.3±10.1|106.5±10.1|
|World|Random|**68±20.8**|2.8±5.5|2.9±4.0|-2.4±2.0|
||Med-Rep|**73.4±23.2**|17.4±8.1|41.5±9.5|8.9±5.1|
||Medium|**93.8±31.8**|73.4±9.3|70.5±15.3|63.9±14.2|
||Expert|**119.7±26.4**|110.3±21.3|107.8±17.7|107.6±15.6|

The results on SMAC are listed below. No results of TD3-BC are provided in SMAC with discrete action space, since it is only suitable for continuous action space.

| Map | Dataset | CFCQL| IQL | AWAC |
| ------- | --------- | ------- | --------- | ---------------- |
|2s3z|Medium| **0.40±0.10** |0.16±0.04|0.19±0.05 |
||Medium_Replay|**0.55±0.07**|0.33±0.06 | 0.39±0.05|
||Expert|**0.99±0.01**|0.98±0.03| 0.97±0.03|
||Mixed|**0.84±0.09**|0.19±0.04| 0.14±0.04| 
|3s_vs_5z|Medium| **0.28±0.03** |0.20±0.05|0.19±0.03 |
||Medium_Replay|**0.12±0.04**|0.04±0.04 | 0.08±0.05|
||Expert|**0.99±0.01**|**0.99±0.01**|**0.99±0.02** |
||Mixed|**0.60±0.14**|0.20±0.06|0.18±0.03 | 
|5m_vs_6m|Medium| **0.29±0.05** |0.25±0.02|0.22±0.04 |
||Medium_Replay|**0.22±0.06**|0.18±0.04 |0.18±0.04 |
||Expert|**0.84±0.03**|0.77±0.03|0.75±0.02 |
||Mixed|0.76±0.07 |0.76±0.06| **0.78±0.02**| 
|6h_vs_8z|Medium| 0.41±0.04|0.40±0.05| **0.43±0.06**|
||Medium_Replay|**0.21±0.05**|0.17±0.03 | 0.14±0.04|
||Expert|**0.7±0.06**|0.67±0.03| 0.67±0.03|
||Mixed|**0.49±0.08** |0.36±0.05|0.35±0.06 | 

We have observed that CFCQL consistently outperforms IQL and AWAC, particularly on the Random dataset. This outcome aligns with our expectations since both IQL and AWAC can be considered as variants of weighted behavior cloning methods, and their performance heavily depends on the quality of the dataset used for training. In contrast, CFCQL does not necessitate the learned policies to closely match the behavior policy, which enables it to excel on datasets with lower quality.

Reference:

[1] Nair, Ashvin, et al. "Awac: Accelerating online reinforcement learning with offline datasets." arXiv preprint arXiv:2006.09359 (2020).


Meta-review: This paper generated a lot of discussion between the reviewers and authors, as well as some internal discussions between the AC and reviewers. Reviewers felt that some of the details of the data composition and corresponding empirical evidence raised some doubts on the efficacy of the algorithm, though it is unclear if this is a flaw: while I think better data compositions could have been utilized, using medium, medium-replay data collection as the authors did is a popular protocol in literature, and I am not aware of any such protocol being utilized with multi-agent systems.

The other concern from reviewers was the presentation -- I agree that the submitted manuscript is not the best in terms of defining terminology and consistency in notation, but I am assuming that the authors will fix this if the paper is accepted. I also think that the paper needs to do a better job at differentiating their contribution and arguing why they attain better performance compared to prior multi-agent offline RL works.

Overall, the paper is borderline, but given that the discussion addressed a number of comments about the paper and since the paper does not deviate from commonly-expected design choices, I am inclining towards accepting this paper. That said, I do think that multi-agent offline RL would benefit from improved benchmarking, and encourage the authors to contribute to such an effort, including open-sourcing and releasing their datasets and environments.
Decision: Accept (poster)