Title: Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-agent reinforcement learning

offline reinforcement learning
Abstract: Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. To mitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent conservative methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

Abstract

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. To mitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.

1 Introduction

Online Reinforcement Learning (Online RL) needs frequently deploying untested policies to environment for data collection and policy optimization, making it dangerous and inefficient to apply in the real-world scenarios (e.g. autonomous vehicle teams). While, Offline Reinforcement Learning (Offline RL) aims to learn policies from a fixed dataset rather than from interacting with the environment, and therefore is suitable for the real applications with highly safety requirements or without efficient simulators [25].

Directly applying off-policy RL to the offline setting may fail due to overestimation [13, 24]. Existing works usually tackle this problem by pessimism. They either utilize behavior regularization to constrain the learning policy close to the behavior policy induced by the dataset [49, 20, 11], or conduct conservative(pessimistic) value iteration to mitigate unexpected overestimation [18, 21]. It has been demonstrated both theoretically and empirically that these methods can achieve comparable performance to their online counterparts under some conditions [21].

Though cooperative Multi-agent Reinforcement Learning (MARL) has gained extraordinary success in various multiplayer video games such as Dota [4], StarCraft [40] and soccer [23], applying current MARL in real scenarios is still challenging due to the same safety and efficiency concerns in singleagent setting, then it is worth conducting investigation for offline RL in multi-agent setting. Compared with single-agent setting, offline RL in multi-agent setting has its own difficulty. On the one hand, the action space explodes exponentially as the agent number increases, then an arbitrary joint action is more likely to be an OOD action given the fixed dataset size, making the OOD phenomenon more severe. This will further exacerbate the issues of extrapolation error and overestimation in the policy evaluation process and thus induce an unexpected or even disastrous final policy [52]. On the other hand, as the core of MARL, the agents need to consider not only their own actions but also other agents' actions as well as contributions to the global return in order to achieve an overall high performance, which undoubtedly increases the difficulty for theoretical analysis. Besides, instead of just guaranteeing single policy improvement, the bounded team performance is also a key concern to offline MARL.

There exist few works to combine MARL with offline RL. Pan et al. [34] uses Independent Learning [44] as a simple solution, i.e., each agent regards others as part of the environment and independently performs offline RL learning. Although mitigating the joint action OOD issue through decoupling the agents's learning completely, it essentially still adopts a single-agent paradigm to learn and thus cannot enjoy the recent progress on MARL that a centralized value function can empower better team coordination [37, 27]. To utilize the advantage of centralized training with decentralized execution (CTDE), Yang et al. [52] applies implicit constraint approach on the value decomposition network [43], which alleviates the extrapolation error and gains some performance improvement empirically. Although it proposes some theoretical analysis for the convergence property of value function, whether the team performance can be bounded from below as agents number increases remains still unknown.

In this paper, we introduce a novel offline MARL algorithm called Counterfactual Conservative Q-Learning (CFCQL) and aim to address the overestimation issue rooted from joint action OOD phenomenon. It adopts CTDE paradigm to realize team coordination, and incorporates the state-ofthe-art offline RL algorithm CQL to conduct conservative value estimation. CQL is preferred due to its theoretical guarantee and flexibility of implementation. One direct treatment is to regard all the agents as a single one and conduct standard CQL learning on the joint policy space which we call MACQL. However, too much conservatism will be generated in this way since the induced penalty can be exponentially large in the joint policy space. Instead, CFCQL separately regularizes each agent in a counterfactual way to avoid too much conservatism. Specifically, each agent separately contributes CQL regularization for the global Q value and then a weighted average is used as an overall regularization. When calculating agent i 's regularization term, rather than sampling OOD actions from the joint action space as MACQL does, we only sample OOD actions for agent i and leave other agents' actions sampled in the dataset. We prove that CFCQL enjoys underestimation property as CQL does and the safe policy improvement guarantee independent of the agents number n , which is advantageous to MACQL especially under the situation with a large n .

We conduct experiments on 1 man-made environment Equal Line, and 3 commonly used multi-agent environments: StarCraft II [40], Multi-agent Particle Environment [27], and Multi-agent MuJoCo [35], including both discrete and continuous action space setting. With datasets collected by Pan et al. [34] and ourselves, our method outperforms existing methods in most settings and even with a large margin on some of them.

We summarize our contributions as follows: (1) we propose a novel offline MARL method CFCQL based on CTDE paradigm to address the overestimation issue and achieve team coordination at the same time. (2) We theoretically compare CFCQL and MACQL to show that CFCQL is advantagous to MACQL on the performance bounds and safe policy improvement guarantee as agent number is large. (3) In hard multi-agent offline tasks with both discrete and continuous action space, our method shows superior performance to the state-of-the-art.

2 Related Works

Offline RL. Standard RL algorithms are especially prone to fail due to erroneous value overestimation induced by the distributional shift between the dataset and the learning policy. Theoretically, it is

proved that pessimism can alleviate overestimation effectively and achieve good performance even with non-perfect data coverage [5, 15, 26, 22, 39, 51, 55, 6]. In the algorithmic line, there are broadly two categories: uncertainty based ones and behavior regularization based ones. Uncertainty based approaches attempt to estimate the epistemic uncertainty of Q-values or dynamics, and then utilize this uncertainty to pessimistically estimating Q in a model-free manner [2, 50, 3], or conduct learning on the pessimistic dynamic model in a model-based manner [54, 16]. Behavior regularization based algorithms constrain the learned policy to lie close to the behavior policy in either explicit or implicit ways [20, 13, 49, 18, 21, 11], and is advantageous over uncertainty based methods in computation efficiency and memory consumption. Among these class of algorithms, CQL[21] is preferred due to its superior empirical performance and flexibility of implementation.

MARL. A popular paradigm for multi-agent reinforcement learning is centralized training with decentralized execution (CTDE) [37, 27]. Centralized training inherits the idea of joint action learning [7], empowering the agents with better team coordination. And decentralized execution enjoys the deploying flexibility of independent learning [44]. In CTDE, some value-based works concentrate on decomposing the single team reward to all agents by value function factorization [43, 37], based on which they further derive and extend the Individual-Global-Max (IGM) principle for policy optimality analysis [42, 47, 38, 46]. Another group of works focus on the actor-critic framework, using a centralized critic to guide each agent's policy update [27, 9]. Some variants of PPO [41], including IPPO [8], MAPPO [53], and HAPPO [19] also show great potential in solving complex tasks. All these works use the online learning paradigm, therefore disturbed by extrapolation error when transferred to the offline setting.

Offline MARL. To make MARL applied in the more practical scenario with safety and training efficiency concerns, offline MARL is proposed. Jiang & Lu [14] shows the challenge of applying BCQ [13] to independent learning, and Pan et al. [34] uses zeroth-order optimization for better coordination among agents' policies. Although it empirically shows fast convergence rate, the policies trained by independent learning have no theoretical guarantee for the team performance. With CTDE paradigm, Yang et al. [52] adopts the same treatment as Peng et al. [36] and Nair et al. [30] to avoid sampling new actions from current policy, and Tseng et al. [45] regards the offline MARL as a sequence modeling problem, solving it by supervised learning. As a result, both methods' performance relies heavily on the data quality. In contrast, CFCQL does not require the learning policies stay close to the behavior policy, and therefore performs well on datasets with low quality. Meanwhile, CFCQL is theoretically guaranteed to be safely improved, which ensures its performance on datasets with high quality.

3 Preliminary

3.1 MARLSymbols

We use a decentralised partially observable Markov decision process (Dec-POMDP) [32] G = ⟨ S, A , I, P, r, Z, O, n, γ ⟩ to model a fully collaborative multi-agent task with n agents, where s ∈ S is the global state. At time step t , each agent i ∈ I ≡ { 1 , ..., n } chooses an action a i ∈ A , forming the joint action a ∈ A ≡ A n . T ( s ′ | s, a ) : S × A × S → [0 , 1] is the environment's state transition distribution. All agents share the same reward function r ( s, a ) : S × A → R . γ ∈ [0 , 1) is the discount factor. Each agent i has its local observations o i ∈ O drawn from the observation function Z ( s, i ) : S × I → O and chooses an action by its policy π i ( a i | o i ) : O → ∆([0 , 1] | A | ) . The agents' joint policy π := ∏ n i =1 π i induces a joint action-value function : Q π ( s, a ) = E [ R | s, a ] , where R = ∑ ∞ t =0 γ t r t is the discounted accumulated team reward. We assume ∀ r, | r | ≤ R max . The goal of MARL is to find the optimal joint policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) , for all π and ( s, a ) ∈ S × A . In offline MARL, we need to learn the optimal π ∗ from a fixed dataset sampled by an unknown behaviour policy β , and we can not deploy any new policy to the environment to get feedback.

3.2 Value Functions in MARL

In MARL settings with discrete action space, we can maintain for each agent a local-observationbased Q function Q i ( o i , a i ) , and define the local policy π i := arg max a Q i ( o i , a ) . To train local policies with single team reward, Rashid et al. [37] proposes QMIX, using a global Q tot to model

the cumulative team reward. Define the Bellman operator T as T Q ( s, a ) = r + γ max a ′ Q ( s ′ , a ′ ) . QMIX aims to minimize the temporal difference:

where ˆ T is the empirical Bellman operator using samples from the replay buffer D , and ˆ Q tot is the empirical global Q function, computed from s and Q i s represented by a neural network satisfying ∂Q tot /∂Q i ≥ 0 . With such restriction, QMIX can ensure the Individual-Global-Max principle that arg max a Q tot ( s, a ) = (arg max a 1 Q 1 ( o 1 , a 1 ) , ..., arg max a n Q n ( o n , a n )) . For continuous action space, a neural network is used to represent the local policy π i for each agent. Lowe et al. [27] proposes MADDPG, maintaining a centralized critic Q tot to directly guide the local policy update by gradient descent. And the update rule of Q tot is similar to Eq. 1, just replacing T with T π . T π Q = r + γP π Q , where P π Q ( s, a ) = E s ′ ∼ T ( ·| s, a ) , a ′ ∼ π ( ·| s ′ ) [ Q ( s ′ , a ′ )] .

3.3 Conservative Q-Learning

Offline RL is prone to the overestimation of value functions. For an intuitive explanation of the underlying cause, please refer to Appendix E.1. Kumar et al. [21] proposes Conservative Q-Learning (CQL), adding a regularizer to the Q function to address the overestimation issue, which maximizes the Q function of the state-action pairs from the dataset distribution β , while penalizing the Q function sampled from a new distribution µ (e.g., current policy π ). The conservative policy evaluation is shown as follows:

(2)

As shown in Theorem 3.2 in Kumar et al. [21], with a large enough α , we can obtain a Q function, whose expectation over actions is lower than that of the real Q, then the overestimation issue can be mitigated.

4 Proposed Method

We first show the overestimation problem of multi-agent value functions in Sec. 4.1, and a direct solution by the naive extension of CQL to multi-agent settings in Sec. 4.2, which we call MACQL. In Sec. 4.3, we propose Counterfactual Conservative Q-Learning (CFCQL) and further demonstrate that it can realize conservatism in a more mild and controllable way which is independent of the number of agents. Then we compare MACQL and CFCQL from the aspect of safe improvement performance. In Sec. 4.4, we present our novel and practical offline RL algorithm.

4.1 Overestimation in Offline MARL

The Q-values in online learning usually face the overestimation problem [1, 33], which becomes more severe in offline settings due to distribution shift and offline policy evaluation [13]. Offline MARL suffers from the same issue even worse since the joint action space explodes exponentially as the agent number increases, then an arbitrary joint action is more likely to be an OOD action given the fixed dataset size. To illustrate this phenomenon, we design a toy MultiAgent Markov Decision Process (MMDP) as shown in Fig.1(a). There are five agents and three states. All agents are randomly initialized and need learn to take three actions

(stay, move up, move down) to move to or stay in the state S 2 . The dataset consists of 1000 samples with reward larger than 0 . 8 r max . As shown in Fig. 1(b), if not specially designed for offline setting,

naive application of QMIX leads to exponentially explosion of value estimates and the sub-optimal policy performance, Fig. 1(c).

4.2 Multi-Agent Conservative Q-Learning

When treating multiple agents as a unified single agent and only regarding the joint policy of the β , µ , π , that is β ( a | s ) = ∏ n i =1 β i ( a i | s ) , µ ( a | s ) = ∏ n i =1 µ i ( a i | s ) and π ( a | s ) = ∏ n i =1 π i ( a i | s ) , we can extend the single-agent CQL to CTDE multi-agent setting straightforwardly and derive the conservative policy evaluation style directly as follows:

where we represent the temporal difference (TD) error concisely as ˆ E D ( π , Q, k ) := 1 2 E s, a ,s ′ ∼D [( Q ( s, a ) -ˆ T π ˆ Q tot k ( s, a )) 2 ] . In the rest of the paper, we mainly focus on the global Q function of a centralized critic and omit the superscript 'tot' for ease of expression.

Similar to CQL, MACQL can learn a lower-bounded Q-value with a proper α , then the overestimation issue in offline MARL can be mitigated. According to Theorem 3.2 in Kumar et al. [21], the degree of pessimism relies heavily on D CQL ( π , β )( s ) := ∑ a π ( a | s )[ π ( a | s ) β ( a | s ) -1] . However, as we will show in Sec. 4.3, D CQL expands exponentially as the number of agents n increases, resulting in an over-pessimistic value function and a mediocre policy improvement guarantee for MACQL.

4.3 Counterfactual Conservative Q-Learning

Intuitively, we aim to learn a value function that prevents the overestimation of the policy value, which is expected not too far away from the true value in the meantime. Similar to MACQL as introduced above, we add a regularization in the policy evaluation process, by penalizing the values for OOD actions and rewarding the values for in-distribution actions. However, rather than regarding all the agents as a single one and conducting standard CQL method in the joint policy space, in CFCQL method, each agent separately contributes regularization for the global Q value and then a weighted average is used as an overall regularization. When calculating agent i 's regularization term, instead of sampling OOD actions from the joint action space as MACQL does, we only sample OOD actions for agent i and leave other agents' actions sampled in the dataset. Specifically, using a -i to denote the joint actions other than agent i . Eq. 4 is our proposed policy evaluation iteration:

where α, λ i are hyper-parameters, and ∑ n i =1 λ i = 1 , λ i ≥ 0 . We refer to our method as 'counterfactual' because its structure bears resemblance to counterfactual methods in MARL [9]. This involves obtaining each agent's counterfactual baseline by marginalizing out a single agent's action while keeping the other agents' actions fixed. The intuitive rationale behind employing a counterfactual-like approach is that by individually penalizing each agent's out-of-distribution (OOD) actions while holding the others' actions constant from the datasets, we can effectively mitigate the out-of-distribution problem in offline MARL with reduced pessimism, as illustrated in the rest of this section.

The theoretical analysis is arranged as follows: In Theorem 4.1 we show the new policy evaluation leads to a milder conservatism on value function, which still lower bounds the true value. Then we compare the conservatism degree between CFCQL and MACQL in Theorem 4.2. In Theorem 4.3 and Theorem 4.4 we show the effect of milder conservatism brought by counterfactual treatments on the performance guarantee (All proofs are defered to Appendix A).

Theorem 4.1 (Equation 4 results in a lower bound of value function) . The value of the policy under the Q function from Equation 4, ˆ V π ( s ) = E π ( a | s ) [ ˆ Q π ( s, a )] , lower-bounds the true value of the policy obtained via exact policy evaluation, V π ( s ) = E π ( a | s ) [ Q π ( s, a )] , when µ = π , according to: ∀ s ∈

D CF CQL ( π , β )( s ) -1 , ∀ s ∈ D , ˆ V π ( s ) ≤ V π ( s ) , with probability ≥ 1 -δ . When T π = ˆ T π , then any α > 0 guarantees ˆ V π ( s ) ≤ V π ( s ) , ∀ s ∈ D .

Theorem 4.1 mainly differs from Theorem 3.2 in Kumar et al. [21] on the specific form of the divergence: D CQL and D CF CQL , both of which determine the degrees of conservatism when π and β are different. Empirically, we find D CQL generally becomes too large when the number of agents expands, resulting in a low, even negative value function. To measure the scale difference between D CQL ( π , β )( s ) := ∏ i ( D CQL ( π i , β i )( s )+1) -1 and D CF CQL ( π , β )( s ) := ∑ i λ i D CQL ( π i , β i )( s ) , we have Theorem 4.2:

Theorem 4.2. ∀ s, π , β , one has 0 ≤ D CF CQL ( π , β )( s ) ≤ D CQL ( π , β )( s ) , and the following inequality holds:

̸

where j = arg max k E π k π k ( s ) β k ( s ) represents the agent whose policy distribution is the most far away from the dataset.

Since D CF CQL can be written as ∑ i λ i D CQL ( π i , β i )( s ) , it can be regarded as a weighted average of each individual agents' policy deviations from its individual behavior policy. Therefore, the scale of D CF CQL is independent of the number of agents. Instead, Theorem 4.2 shows that both D CQL and the ratio of D CQL and D CF CQL explode exponentially as the number of agents n increases.

Then we discuss the influence of D CF CQL on the property of safe policy improvement. Define ˆ M as the MDP induced by the transitions observed in the dataset. Let the empirical return J ( π , ˆ M ) be the discounted return for any policy π in ˆ M and ˆ Q be the fixed point of Equation 4. Then the optimal policy of CFCQL is equivalently obtained by solving:

See the proof in Appendix A.3. Eq. 6 can be regarded as adding a D CF CQL -related regularizer to J ( π , ˆ M ) . Replacing D CF CQL with D CQL in the regularizer, we can get a similar form of π ∗ MA for MACQL. We next discuss the performance bound of CFCQL and MACQL on the true MDP M . If we assume the full coverage of β i on M , we have:

Theorem 4.3. Assume ∀ s, a, i, β i ( a | s ) ≥ ϵ , then with probability ≥ 1 -δ ,

where π ∗ = arg max π J ( π , M ) , i.e. the optimal policy, and sampling error is a constant dependent on the MDP itself and D .

Theorem 4.3 shows that when sampling error and α are small enough, the performances gap induced by π ∗ CF can be small, but that induced by π ∗ MA expands when n increases, making π ∗ MA far away from the optimal. Upon examining the performance gap, we ultimately compare the safe policy improvement guarantees of the two methods on M :

Theorem 4.4. Assume ∀ s, a, i, β i ( a | s ) ≥ ϵ . The policy π ∗ MA ( a | s ) is a ζ MA -safe policy improvement over β in the actual MDP M , i.e., J ( π ∗ MA , M ) ≥ J ( β , M ) -ζ MA . And the policy π ∗ CF ( a | s ) is a ζ CF -safe policy improvement over β in M , i.e., J ( π ∗ CF , M ) ≥ J ( β , M ) -ζ CF . When n ≥ log 1 ϵ ( 1 ϵ + 2 α √ | A | √ |D ( s ) | ( C r,δ + γR max C T,δ 1 -γ ) · ( 1 √ ϵ -1) ) , ζ CF ≤ ζ MA .

Detailed formation of ζ MA and ζ CF is provided in Appendix A.5. Theorem 4.4 shows that with a large enough agent count n , CFCQL has better safe policy improvement guarantee than MACQL. The validation experiment of this theoretical result is presented in Section 5.1.

4.4 Practical Algorithm

The fixed point of Eq. 4 provides an underestimated Q function for any policy µ . But it is computationally inefficient to solve Eq. 4 every time after one step policy update. Similar to CQL [21], we also choose a µ ( a | s ) that would maximize the current ˆ Q with a µ -regularizer. If the regularizer aims to minimize the KL divergence between µ i and a uniform distribution, µ i ( a i | s ) ∝ exp ( E a -i ∼ β -i Q ( s, a i , a -i ) ) , which results in the update rule of Eq. 7 (See Appendix B.1 for detailed derivation):

Finally, we need to specify each agent's weight of minimizing the policy Q function λ i . Theoretically, any simplex of λ that satisfies ∑ n i =1 λ i = 1 can be used to induce an underestimated value function linearly increasing as agents number as we expect. Therefore, a simple way is to set λ i = 1 n , ∀ i where each agent contributes penalty equally. Another way is to prioritize penalizing the agent that exhibits the greatest deviation from the dataset, which is the one-hot style of λ :

We assert that each agent's conservative contribution deserves to be considered and differed according to their degree of deviation. As a result, both the uniform and the one-hot treatment present some limitations. Consequently, we employ a moderate softmax variant of Eq. 8:

where τ is a predefined temperature coefficient, controlling the influence of E π i π i β i on λ i . When τ → 0 , λ i → 1 n , and when τ → ∞ , it turns into Eq. 8. To compute Eq. 9, we need an explicit expression of π i and β i . In discrete action space, π i can be estimated by exp ( E a -i ∼ β -i Q ( s, a i , a -i ) ) , and we use behavior cloning [29] to train a parameterized β ( s ) from the dataset. In continuous action space, π i is parameterized by each agent's local policy. For β i , we use the method of explicit estimation of behavior density in Wu et al. [48], which is modified from a VAE [17] estimator. Details for computing λ are defered to Appendix B.2.

For policy improvement in continuous action space, we also take derivation of a counterfactual Q function for each agent, rather than updating all agents' policy together like in MADDPG. Specifically, the gradient of each agent i 's policy π i is calculated by:

The reason is that in CFCQL, we only minimize Q ( s, π i , β -i ) , rather than Q ( s, π ) . Using the untrained Q ( s, π ) to directly guide PI like MADDPG may result in a bad policy.

We summarize CFCQL in discrete and continuous action space in Algorithm 1 as CFCQL-D and -C, separately.

Algorithm 1 CFCQL-D and CFCQL-C

1: Initialize Q θ , target network Q ˆ θ , target update interval t tar , replay buffer D , and optionally π ψ for CFCQL-C

2: for t = 1 , 2 , . . . , t max do

3: Sample N transitions { s, a , s ′ , r } from D

4: Compute Q θ ( s, a ) using the structure of QMIX for CFCQL-D or MADDPG for CFCQL-C.

5: Calculate λ according to Eq. 9

6: Update Q θ by Eq. 7 with sampled transitions. Using ˆ T ˆ θ for CFCQL-D, and ˆ T π ψt ˆ θ for CFCQL-C

7: (Only for CFCQL-C) For each agent i , take one-step policy improvement for π i ψ according to Eq. 10

8: if t mod t tar = 0 then

9: update target network ˆ θ ← θ

10: end if

11: end for

5 Experiments

Baselines. We compare our method CFCQL with several offline Multi-Agent baselines, where baselines with prefix MA adopt CTDE paradigm and the others adopt independent learning paradigm: BC : Behavior cloning. TD3-BC [11]: One of the state-of-the-art single agent offline algorithm, simply adding the BC term to TD3 [12]. MACQL : Naive extension of conservative Q-learning, as proposed in Sec.4.2 . MAICQ [52]: Multi-agent version of implicit constraint Q-learning by decomposed multi-agent joint-policy under implicit constraint. OMAR [34]: Using zeroth-order optimization for better coordination among agents' policies, based on independent CQL ( ICQL ). MADTKD [45]: Using decision transformer to represent each agent's policy, trained with knowledge distillation. IQL [18] and AWAC [31]: variants of advantage weighted behaviour cloning, which are SOTA on single agent offline RL. Details for baseline implementations are in Appendix C.3.

Each algorithm is run for five random seeds, and we report the mean performance with standard deviation 2 . Four environments are adopted to evaluate our method, including both discrete action space and continuous action space. We have relocated additional experimental results to Appendix D to conserve space.

5.1 Equal Line

To empirically compare CFCQL and MACQL as agents number n increases, we design a multi-agent task called Equal _ Line , which is a one-dimensional simplified version of Equal _ Space introduced in Tseng et al. [45].

Figure 2: (a) The Equal _ Line environment where n=3. (b) Performance ratio of CFCQL and MACQL to the behaviour policy with a varing number of agents.

Details of the environment are in Appendix C.1. The n agents need to cooperate to disperse and ensure every agent is equally spaced. The datasets consist of 1000 trajectories sampled by executing a fully-pretrained policy of QMIX[37], i.e. Expert dataset. We plot the performance ratios of CFCQL and MACQL to the behavior policy for different agent number n in Fig.2(b). It can be observed that the performance of MACQL degrades dramatically as the increase of number of agents while the performance of CFCQL remains basically stable. The results strongly verify the conclusion we proposed in Sec.4.3, that CFCQL has better policy improvement guarantee than MACQL with a large enough number of agents n .

5.2 Multi-agent Particle Environment

In this section, we test CFCQL on Multi-agent Particle Environemnt with continuous action space. We use the dataset and the adversary agent provided by Pan et al. [34]. The performance of the trained model is measured by the normalized score 100 × ( S -S Random ) / ( S -S Expert ) [10].

In Table 1, we only show the comparison of our method and the current state-of-theart method OMAR and IQL to save space. For complete comparison with more baselines, e.g., TD3+BC and

AWAC, please refer to Appendx D.1. It can be seen that on 11 of the 12 datasets, CFCQL shows superior performance than current state-of-the-art. Note that we only report the results of τ = 0 . In Appendix D.2 of the ablation on τ , we show that with carefully fine-tuned τ , higher scores of CFCQL can be obtained. We also carry out ablations of α in Appendix D.3.

5.3 Multi-agent MuJoCo

In this section we investigate the effect of our method on more complex continuous control task. We use the HalfCheetah-v2 setting from the multi-agent MuJoCo environment [35] and the datasets provided by Pan et al. [34]. Table 2 shows that CFCQL exceeds the current state-of-theart on most datasets.

Except for the counterfactual Q function, we also analyze whether the counterfactual treatment in CFCQL can be incorporated in other components and help further improvement in Appendix D.4.

We find that the counterfactual policy improvement, i.e., the policy improvement by Eq. 10 rather than using MADDPG's PI, is critical for our method.

5.4 StarCraft II

We further validate CFCQL's universality through complex experiments on the StarCraft II Micromanagement Benchmark [40], encompassing four maps with varying agent counts and difficulties: 2s3z, 3s_vs_5z, 5m_vs_6m, and 6h_vs_8z. As no pre-existing datasets exist for these tasks, we generate them ourselves, with dataset creation details provided in the Appendix C.2.

Table 3 presents the average test winning rates of various algorithms on different datasets. MACQL's perfor-

0.8

rate

0.6

Winning

0.4

50.2

3

0.10.20.5110co00.10.20.5110co00.10.20.51100o00.10.20.5110oo

TemperatureCoefficient/t

(a)Med-Replay

(b) Medium

(c) Mixed

(d) Expert

mance depends on agent count and environmental difficulty, only succeeding in 2s3z and 3s_vs_5z. MAICQ and OMAR perform well across many datasets but can not tackle all tasks and significantly

struggle in 6h_vs_8z, a highly challenging map. MADTKD, employing supervised learning and knowledge distillation, works well but seldom surpasses BC. IQL and AWAC are competitive baselines but they still fall short compared to CFCQL in most datasets. CFCQL significantly outperforms all baselines on most datasets, achieving state-of-the-art results, with its success attributed to moderate and appropriate conservatism compared to MACQL and other baselines.

Temperature Coefficient. To study the effect of temperature coefficient τ , the key hyperparameter in computing the λ i , Fig.3(a) shows the testing winning rate of CFCQL with different τ s on each kind of dataset of map 6h_vs_8z. As shown, CFCQL is not sensitive to this hyperparameter and we find that the best value of τ is usually greater than 0 while lower than infinity, showing that moderate imbalance can be more effective as expected in previous section.

6 Discussion and Conclusion

6.1 Broader Impacts

Our proposed method holds potential for application in real-world multi-agent systems, such as intelligent warehouse management or medical treatment. However, directly implementing the derived policy might entail risks due to the domain gap between the training virtual datasets and real-world scenarios. To mitigate potential hazards, it is crucial for practitioners to operate the policy under human supervision, ensuring that undesirable outcomes are avoided by limiting the available options.

6.2 Limitations

Here we discuss some limitations about CFCQL. In the case of discrete action space, since CFCQL uses QMIX as the backbone, it inherits the Individual-global-max principle [42], which means it cannot solve tasks that are not factorizable. On continuous action space, the counterfactual policy update used in CFCQL allows for updating only one agent's policy for each sample, which may lead to lower convergence speed compared to methods with independent learning.

6.3 Conclusion

In this paper, we study the offline MARL problem which is practical and challenging but lack of enough attention. We demonstrate from theories and experiments that naively extending conservative offline RL algorithm to multi-agent setting leads to over-pessimism as the exponentially explosion of action space with the increasing number of agents which hurts the performance. To address this issue, we propose a counterfactual way to make conservative value estimation while maintaining the CTDE paradigm. It is theoretically proved that the proposed CFCQL enjoys performance guarantees independent of number of agents, which can avoid overpessimism caused by the exponentially growth of joint policy space. The results of experiments with discrete or continuous action space show that our method achieve superior performance to the state-of-the-art. Some ablation study is also made to propose further understanding of CFCQL. The idea of counterfactual treatment may also be incorporated with other offline RL and multi-agent RL algorithms, which deserves further investigation both theoretically and empirically.

References

Ackermann, J., Gabler, V., Osa, T., and Sugiyama, M. Reducing overestimation bias in multiagent domains using double centralized critics. arXiv preprint arXiv:1910.01465 , 2019.

Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pp. 104-114. PMLR, 2020.

An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems , 34, 2021.

Berner, C., Brockman, G., Chan, B., Cheung, V., D˛ ebiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Buckman, J., Gelada, C., and Bellemare, M. G. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799 , 2020.

Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.02446 , 2022.

Claus, C. and Boutilier, C. The dynamics of reinforcement learning in cooperative multiagent systems. In AAAI Conference on Artificial Intelligence (AAAI) , 1998.

de Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H., Sun, M., and Whiteson, S. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 , 2020.

Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI) , 2018.

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pp. 1587-1596. PMLR, 2018.

Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pp. 2052-2062. PMLR, 2019.

Jiang, J. and Lu, Z. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108.01832 , 2021.

Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pp. 5084-5096. PMLR, 2021.

Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05951 , 2020.

Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pp. 5774-5783. PMLR, 2021.

Kuba, J. G., Chen, R., Wen, M., Wen, Y., Sun, F., Wang, J., and Yang, Y. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251 , 2021.

Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Kumar, A., Hong, J., Singh, A., and Levine, S. Should i run offline reinforcement learning or behavioral cloning? In Deep RL Workshop NeurIPS 2021 , 2021.

Kurach, K., Raichuk, A., Sta´ nczyk, P., Zaj ˛ ac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., et al. Google research football: A novel reinforcement learning environment. arXiv preprint arXiv:1907.11180 , 2019.

Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning , pp. 1702-1712. PMLR, 2022.

Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08202 , 2020.

Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Abbeel, O. P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in neural information processing systems (NeurIPS) , 2017.

Meng, L., Wen, M., Yang, Y., Le, C., Li, X., Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv:2112.02845 , 2021.

Michie, D., Bain, M., and Hayes-Miches, J. Cognitive models from subcognitive skills. IEE control engineering series , 44:71-99, 1990.

Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Oliehoek, F. A., Amato, C., et al. A concise introduction to decentralized POMDPs . Springer, 2016.

Pan, L., Rashid, T., Peng, B., Huang, L., and Whiteson, S. Regularized softmax deep multi-agent q-learning. Advances in Neural Information Processing Systems , 34:1365-1377, 2021.

Pan, L., Huang, L., Ma, T., and Xu, H. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International Conference on Machine Learning , pp. 17221-17237. PMLR, 2022.

Peng, B., Rashid, T., Schroeder de Witt, C., Kamienny, P.-A., Torr, P., Böhmer, W., and Whiteson, S. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems , 34:12208-12221, 2021.

Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485 , 2018.

Rashid, T., Farquhar, G., Peng, B., and Whiteson, S. Weighted qmix: Expanding monotonic value function factorisation. arXiv e-prints , pp. arXiv-2006, 2020.

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34, 2021.

Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043 , 2019.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (ICML) , 2019.

Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V. F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) , 2018.

Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning (ICML) , 1993.

Tseng, W.-C., Wang, T.-H., Lin, Y.-C., and Isola, P. Offline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Processing Systems , 2022.

Wan, L., Liu, Z., Chen, X., Wang, H., and Lan, X. Greedy-based value representation for optimal coordination in multi-agent reinforcement learning. arXiv preprint arXiv:2112.04454 , 2021.

Wang, J., Ren, Z., Liu, T., Yu, Y ., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062 , 2020.

Wu, J., Wu, H., Qiu, Z., Wang, J., and Long, M. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., and Goh, H. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140 , 2021.

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34, 2021.

Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:10299-10312, 2021.

Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 , 2021.

Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239 , 2020.

Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34, 2021.

A Detailed Proof

A.1 Proof of Theorem 4.1

Proof. Similar to the proof of Theorem 3.2 in Kumar et al. [21], we first prove this theorem in the absence of sampling error, and then incorporate sampling error at the end. By set the derivation of the objective in Eq. 4 to zero, we can compute the Q-function update induced in the exact, tabular setting( T π = ˆ T π and π β ( a | s ) = ˆ π β ( a | s ) ).

Then, the value of the policy, ˆ V k +1 can be proved to be underestimated, since:

Next, we will show that D CF CQL ( s ) = Σ a π ( a | s ) [ Σ n i =1 λ i µ i ( a i | s ) ˆ π i β ( a i | s ) -1 ] is always positive, when µ i ( a i | s ) = π i ( a i | s ) :

As shown above, the D CF CQL ( s ) ≥ 0 , and D CF CQL ( s ) = 0 , iff π i ( a i | s ) = π i β ( a i | s ) . This implies that each value iterate incurs some underestimation, i.e. ˆ V k +1 ( s ) ≤ T π ˆ V k ( s ) .

We can compute the fixed point of the recursion in Equation A.2 and get the following estimated policy value:

Because the ( I -γP π ) -1 is non negative and the D CF CQL ( s ) ≥ 0 , it's easily to prove that in the absence of sampling error, Theorem 4.1 gives a lower bound.

Incorporating sampling error . According to the conclusion in Kumar et al. [21], we can directly write down the result with sampling error as follows:

So, the statement of Theorem 4.1 with sampling error is proved. Please refer to the Sec.D.3 in Kumar et al. [21] For detailed proof. Besides, the choice of α in this case to prevent overestimation is given by:

A.2 Proof of Theorem 4.2

Proof. According to the definition, we can get the formulation of D CF CQL ( π , β )( s ) and D CQL ( π , β )( s ) as follow:

Then, by taking the logarithm of D CQL ( π , β )( s ) , we get:

As Σ i λ i = 1 , it's obvious that

By combining equation A.16 and inequation A.17, we get

̸

̸

the second inequality is derived from the Jensen's inequality. As the Kullback-Leibler Divergence is non-negative, it's obvious that D CQL ( π , β )( s ) ≥ D CF CQL ( π , β )( s ) , then we can simplify the left-hand side of this inequality:

̸

A.3 Proof of Equation 6

Proof. Similar to the proof of Lemma D.3.1 in CQL [21], Q is obtained by solving a recursive Bellman fixed point equation in the empirical MDP ˆ M , with an altered reward, r ( s, a ) -α [ ∑ i λ i π i ( a i | s ) β i ( a i | s ) -1 ] , hence the optimal policy π ∗ ( a | s ) obtained by optimizing the value under the CFCQL Q-function equivalently is characterized via Eq. 6.

A.4 Proof of Theorem 4.3

Proof. Similar to Eq. 6, π ∗ MA is equivalently obtained by solving:

Recall that ∀ s, π , β , D CQL ( π , β )( s ) ≥ 0 . We have

Then we give an upper bound of E s ∼ d π ∗ ˆ M ( s ) [ D CQL ( π ∗ , β )( s )] . Due to the assumption that β i is greater than ϵ anywhere, we have

Combining Eq. A.22 and Eq. A.23, we can get

Recall the sampling error proved in [21] and referred to above in (A.10), we can use it to bound the performance difference for any π on true and empirical MDP by

then let sampling error := 2 · C r,T,σ R max (1 -γ ) 2 ∑ s ρ ( s ) √ | D ( s ) | , and incorporate it into (A.24) , we get

where sampling error is a constant dependent on the MDP itself and D. Note that during the proof we do not take advantage of the nature of π ∗ . Actually π ∗ can be replaced by any policy π . The reason we use π ∗ is that it can give that largest lower bound, resulting in the best policy improvement guarantee. Similarly, D CF CQL can be bounded by 1 ϵ -1 :

A.5 Proof of Theorem 4.4

We first show the theorem of safe policy improvement guarantee for MACQL and CFCQL, separately. Then we compare these two gaps.

MACQL has a safe policy improvement guarantee related to the number of agents n :

Proof. We can first get a J ( π ∗ MA , ˆ M ) -related policy improvement guarantee following the proof of Theorem 3.6 in Kumar et al. [21]:

According to Eq. A.21, π ∗ MA is obtained by optimizing J ( π , ˆ M ) with a D CQL -related regularizer. And Theorem 4.3 shows that D CQL can be extremely large when the team size expands, which may severely change the optimization objective and affects the shape of the optimization plane. Therefore, J ( π ∗ MA , ˆ M ) may be extremely low, and keeping J ( π ∗ MA , ˆ M ) in Eq. A.28 results in a mediocre policy improvement guarantee. To bound J ( π ∗ MA , ˆ M ) , we introduce Eq. A.24 into Eq. A.28, we get the following:

This complete the proof.

Wecan get a similar ζ CF satisfying J ( π ∗ CF , M ) ≥ J ( β , M ) -ζ CF for CFCQL, which is independent of n :

Then we can prove Theorem 4.4.

Proof. Subtract ζ CF from ζ MA , and we get:

Let the right side ≥ 0 , and we can get

According to Theorem 4.3,

In the meantime, we have

Therefore, we can relax the lower bound of n to a constant that

B Implement Details

B.1 Derivation of the Update Rule

To utilize the Eq. 4 for policy optimization, following the analysis in the Section 3.2 in Kumar et al. [21], we formally define optimization problems over each µ i ( a i | s ) by adding a regularizer R ( µ i ) . As shown below, we mark the modifications from the Eq. 4 in red.

By choosing different regularizer, there are a variety of instances within CQL family. As recommended in Kumar et al. [21], we choose R ( µ i ) to be the KL-divergence against a Uniform distribution over action space, i.e., R ( µ i ) = -D KL ( µ i , Unif ( a i )) . Then we can get the following objective for µ i :

where ∀ s, f ( x ) = Q ( s, x, a -i ) . The optimal solution is:

where Z is the normalization factor, i.e., Z = ∑ x exp( f ( x )) . Plugging this back into Eq. B.36, we get:

B.2 Details for Computing λ

To compute λ , we need an explicit expression of π i and β i . In the setting of discrete action space, as we use Q-learning, π i can be expressed by the Boltzman policy, i.e.

Weuse behaviour cloning to pre-train a parameterized β ( s ) with a three-level fully-connected network and MLE(Maximum Likelihood Estimation) loss.

With the explicit expression of π i and β i , we can directly compute λ with Eq. 8 and Eq. 9. While, in practice, we find the E π i π i ( s ) β i ( s ) may introduce extreme variance as its large scale and fluctuations,

which will hurt the performance. Instead, we take the logarithm of it and further reduced it to the Kullback-Leibler Divergence as follow:

For continuous action space, we use the deterministic policy like in MADDPG, whose policy distribution can be regared as a Dirac delta function. Therefore, we approximate E π j π j ( s ) β j ( s ) by the following:

Then we need to obtain an explicit expression of β i . We first train a V AE [17] from the dataset to obtain the lower bound of β i . Let p ϕ ( a, z | s ) and q φ ( z | a, s ) be the decoder and the encoder of the trained VAE, respectively. According to Wu et al. [48], β j ( a j | s ) can be explicitly estimated by (We omit the superscript j for brevity):

Therefore, we can sample from the VAE L times to estimate β i . The sampling error reduces as L increases.

C Experimental Details

C.1 Tasks

Equal _ Line is a multi-agent task which we design by simplify the space shape of Equal _ Space to one-dimension. There are n agents and they are randomly initialized to the interval [0 , 2] . The state space is a a one-dimensional bounded region in [0 , max(10 , 2 ∗ n )] and the local action space is a discrete, eleven-dimensional space, i.e. [0 , -0 . 01 , -0 . 05 , -0 . 1 , -0 . 5 , -1 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1] , which represents the moving direction and distance at each step. The reward is shared by the agents and formulated as 10 ∗ ( n -1) min _ dis -last _ step _ min _ dis line _ length , which will spur the agents to cooperate to spread out and keep the same distance between each other.

For Multi-agent Particle Environment and Multi-agent Mujoco, we adopt the open-source implementations from Lowe et al. [27] 3 and Peng et al. [35] 4 respectively. And we use the datasets and the adversary agents provided by Pan et al. [34].

For StarCraft II Micromanagement Benchmark, we use the open-source implementation from Samvelyan et al. [40] 5 and choose four maps with different difficulty and number of agents as the experimental scenarios, which is summarized in Table 4. We construct our own datasets with QMIX [37] by collecting training or evaluating data.

C.2 StarCraft II datasets collection

The datasets are made based on the training process or trained model of QMIX[37]. Specially, the Medium or Expert datasets are sampled by executing a partially-pretrained policy with a medium performance level or a fully-pretrained policy. The Medium -Replay datasets are exactly the replay buffer during training until the policy reaches the medium performance. The Mixed datasets are the equal mixture of Medium and Expert datasets. All datasets contain five thousand trajectories, except for the Medium -Replay .

C.3 Baselines

BC : behavior cloning. In discrete action space, we train a three-level MLP network with MLE loss. In continuous action space, we use the method of explicit estimation of behavior density in Wu et al. [48], which is modified from a VAE [17] estimator. TD3-BC [11]: One of the SOTA single agent offline algorithm, simply adding the BC term to TD3 [12]. We use the open-source implementation 6 and modify it to a CTDE version with centralised critic. IQL [18] and AWAC [31]: variants of advantage weighted behaviour cloning. We refer to the open-source implementation 7 and implement a CTDE version similar to TD3-BC. MACQL :naive extension of conservative Q-learning, as proposed in Sec. 3.3. We implement it based on the open-source implementation 8 . As the joint action space is enormous, we sample N actions for the logsumexp operation. MAICQ [52]:multi-agent version of implicit constraint Q-learning by propose the decomposed multi-agent joint-policy under implicit constraint. We use the open-source implementation 9 in discrete action space and cite the experimental results in continuous action space from Pan et al. [34]. OMAR [34]:uses zeroth-order optimization for better coordination among agents' policies, based on independent CQL ( ICQL ). We cite the experimental results in continuous action space from Pan et al. [34] and implement a version in discrete action space based on the open-source implementation 10 . MADTKD [45]:uses decision transformer to represent each agent's policy and trains with knowledge distillation. As lack of open-source implementation, We implement it based on the open-source implementation 11 of another Decision Transformer based method MADT [28].

C.4 Resources

We use 2 servers to run all the experiments. Each one has 8*NVIDIA RTX 3090 GPUs, and 2*AMD 7H12 CPUs. Each setting is repeated for 5 seeds. For one seed in SC2, it takes about 1 . 5 hours. For MPE, 10 minutes is enough. The experiments on MaMuJoCo cost the most, about 5 hours for each seed.

C.5 Code, Hyper-parameters and Reproducibility

Please refer to this repository 12 for the code, datasets and the hyper-parameters of our method. For each dataset number 0 , 1 , 2 , 3 , 4 , we use the seed 0 , 1 , 2 , 3 , 4 , respectively.

D More results

D.1 Complete Results on MPE

Table 5 shows the complete results of our methods and more baselines on Multi-agent Particle Environment. Some results are cited from Pan et al. [34].

Random

Med-Rep

Medium

Expert

120

160

100

140

140

100

120

80

120

80

100

100

Score

60

core

60

80

80

40

40

60

60

40

40

20

20

20

20

Q

0.010.10.5

1

2

5

10

50

00

Q

0

0.010.10.5

1

2

5

10

50

00

Q

0

0.010.10.5

1

2

5

10

50

00

0

0

0.010.10.5

1

2

5

10

50

00

T

1

T

1

D.2 Temperature Coefficient in Continuous Action Space

We carry out ablations of τ on MPE's map World in Fig. 4. We find that although. the best τ differs in different datasets, the overall performance is not sensitive to τ , which verifies the theoretical analysis that any simplex of λ that ∑ n i =1 λ i = 1 can induce an underestimated value function.

D.3 Ablation on CQL α

We carry out ablations of α on MPE's map World in Fig. 5. We find that α plays a more important role for team performance on narrow distributions (e.g., Expert and Medium ) than that on wide distributions (e.g., Random and Medium -Replay ).

D.4 Component Analysis on Counterfactual style

In the environment MaMuJo, except for the counterfactual Q function, we also analyze whether the conuterfactual treatment in CFCQL can be incorporated in other components and help further improvement in Table 6. We find that the counterfactual policy improvement is critical for this

Random

Med-Rep

Medium

Expert

120

140

100

100

120

150

80

80

100

125

Score

60

Score

60

Score

80

60

core

100

75

40

40

40

50

20

20

25

20

0

0

0

-20

0.2

0.30.40.50.60.70.81.01.21.5

0.1

0.2

0.30.40.50.60.70.80.91.0

0.1

0.2

0.30.40.50.60.70.80.91.0

0.1

0.20.30.50.70.91.21.51.82.1

α

α

α

α

environment. With CF_P, the method shows great performance gain on narrow data distribution, e.g., the Expert dataset.

0.8

■50500

5000

50000

rate

0.6

Test Winning

0.4

0.2

0

Medium

Mixed

Expert

Size of the DataSet

D.5 Analysis on Size of Dataset

An additional study examines dataset size effects. We generate a full dataset with 50 , 000 trajectories for each type in map 6 h _ vs _ 8 z , creating smaller datasets by sampling 5 , 50 , 500 , 5000 trajectories. Fig. 6 displays CFCQL's testing winning rate for varying dataset sizes. It's notable that to ensure fairness, the maximum number of training steps for all datasets and algorithms on the SMAC environment is fixed at 1e7. In this additional study, however, we trained the CFCQL algorithm until convergence to eliminate the impact of underfitting. The results demonstrate that larger datasets contribute to improved convergence performances, thus confirming the scalability of CFCQL for larger data samples.

E Discussion

E.1 Overestimation in Offline RL

We offer an intuitive explanation for the phenomenon of overestimation caused by distribution shift in this section. For a rigorous proof, we refer readers to the related works.

In offline RL, a key challenge arises due to the distribution mismatch between the behavior policy-the policy responsible for generating the data-and the target policy, which is the policy one aims to improve. This mismatch can result in extrapolation errors during value function estimation, often leading to overestimation. Specifically, during the policy evaluation stage, the dataset may not encompass all possible state-action pairs in the Markov Decision Process (MDP), leading to inaccurate Q function estimates for unseen state-action pairs. These estimates may be either too high or too low compared to the actual Q values. Subsequently, in the policy improvement stage, the algorithm tends to shift towards actions that appear to offer higher rewards based on these overestimated values. Unlike online RL, which can implement the current training policy in the environment to obtain real feedback and thereby correct the policy's direction, offline RL lacks this corrective mechanism unless careful loss design is employed. When using common bootstrapping methods like temporal difference learning for training the Q function, these overestimation errors can propagate, affecting estimates

for other state-action pairs. This can result in a chain reaction of overestimations, potentially causing an exponential explosion of the Q function.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of offline multi-agent reinforcement learning. A way to make conservative updates to the Q-function is proposed, extending, non-trivially, CQL to multi-agent settings. Theory and experiments clearly validate the approach.\n\nSTRENGTHS:\nThe method is both well motivated as well as theoretically and empirically validated. The method is experimentally shown to be substantially better than all the state-of-the-art baselines considered.\n\nWEAKNESSES:\nThe introduction can be improved in terms of written language. Moreover, I would like to see a couple of straightforward independent learning baselines added to the StarCraft results. See my questions below.\n\nMinor comments:\n- I do not understand the last sentence of 3.3;\n- Some sentences to improve: figure 1(a), first sentence; first paragraph of the introduction;\n- I suggest giving an intuitive meaning to Theorems 1 and 2 after they appear, as well as an intuitive meaning of the whole theory of Section 4.3 in the end of the Section.\n- typos: \"temporature\" in the caption of Figure 3(a); capitalize \"conclusion\" in the title of Section 6.\n\nQUESTIONS:\n- In Section 4.1, what are the optimal Q-values of the Toy MDP ?\n- What is $\\beta$ ? It appears first in Section 3.3 and I don't see it defined, even though it seems quite important in the sequel (for instance, it appears twice in the proposed algorithm).\n- Why is the method called counterfactual ? I suggest adding an intuitive explanation to the document.\n- What is the performance of single agent independent Q-learning (actually DQN) and of independent CQL (actually DQN) on the SMAC environment ? I believe it would be quite important to make clear on the table of the results that coordination is really necessary to perform the task and that ICQL is not enough.\n\nLIMITATIONS:\nI can not see the limitations of the work nor I foresee a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOffline Reinforcement Learning in the multi-agent setting suffers from the combined effects of distribution shift and increasing number of agents. An exponential blowup of the action space in addition to Out-Of Distribution (OOD) actions hinders performance of RL agents. The work tackles these phenomena by proposing CounterFactual Conservative Q Learning (CFCQL), an algorithm that learns conservatively from static offline datasets. CFCQL differs from the naive multi-agent CQL approach as it conservatively trains each agent separately using the pessimistic regularization. Regularization terms are then linearly combined for global value estimation. This prevents agents from being excessively conservative while maintaining theoretical guarantees of underestimation and safe policy improvement as in CQL. In practice, weighted contributions of conservative penalties are realized either using a one-hot encoding or softmax with temperature scaling. In the case of continuous actions, counterfactual Q function updates of the gradient are utilized. Experiments are carried out on a range of discrete and continuous multi-agent benchmarks with promising improvements.\n\nSTRENGTHS:\n* The paper presents an amenable combination of design choices.\n* Empirical results are promising and extensive.\n\nWEAKNESSES:\n* **Writing and Presentation**: My main concern is the writing and presentation of the paper. The paper is not well written and presented in an unorganized manner. Technical claims made by authors are vague and informal. Explanations are not well supported by intuition or insights and the frequency of grammatical errors is too high. Specifically, sections 1 and 2 motivate the work with high-level and vague statements. Section 3.3 does not formally explain the offline RL problem (of maximizing the expected discounted return), behavior policy or distribution shift. Section 4 builds the algorithm using informal vocabulary. Section 4.4 presents theoretical guarantees without any intuition or interpretation and section 6 summarizes the paper informally. In my view, the paper's presentation requires significant attention.\n* **Dataset Ablations**: I am having trouble understanding results on dataset ablations presented in figure 3 (b). Ideally, as the number of data samples grow, the performance of agents improves. With larger static datasets agents have access to more in-distribution samples and a broader coverage of the underlying MDP. However, figure 3 (b) demonstrates that performance of agents drops for 50,000 samples. It would be helpful if the authors can explain this result or its interpretation. In its current form, it appears that the approach may not scale well to larger multi-agent learning datasets for real-world applications.\n* **Baselines**: While the paper includes relevant multi-agent learning baselines, it is worth noting that none of these baselines were designed for offline RL. All methods were developed as off-policy or pure online RL methods that aggregate new experience. With that said, the only baseline of interest is MACQL since it leverages the CQL penalty designed for static datasets. Authors should consider offline-RL baselines that may be adapted for multi-agent learning. For instance, IQL[1] and TD3-BC[2] are state-of-the-art offline RL methods which might be useful for comparison. Similarly, AWR[3] is another method that imitates dataset interactions.\n* **Transfer and Finetuning**: The paper claims that CFCQL addresses distribution shift and generalizes better. However, this claim is not well supported. Ideally, robustness and generalization ability of an algorithm are tested by transferring it to new unseen scenarios. Authors should consider finetuning agents or adapting them to a new task even if on a small toy example. This will help verify the claims of CFCQL addressing distribution shift.\n\n\nWhile experiments and results are promising, overall presentation and writing of the paper needs significant improvement.\n\n## Minors\n\n* line 17: man-made -> synthetic\n* line 23: ~from~\n* line 24: ~the~\n* line 24: highly -> high\n* line 37: agent number -> number of agents\n* line 44: ~just~\n* line 55: bounded from below -> lower bounded\n* line 63: too much -> excessive\n* line 66: contributes CQL -> contributes to CQL\n* line 69: sampled in the dataset -> sampled in-distribution\n* line 70: agents number -> number of agents\n* line 72: man-made -> synthetic\n* line 79: advantagous -> advantageous\n* line 149: dataset distribution -> behavior policy\n* Figure 1: stays -> stay\n* line 171: even worse -> more significantly\n* line 175: need learn -> need to learn\n* line 178: exponentially -> exponential\n* line 183: style -> update\n* line 198: contributes regularization -> contributes its regularization\n* line 201: in the dataset -> in-distribution\n* line 238: performances -> performance\n* line 257: contributes penalty -> contributes to penalty\n* line 258: style -> encoding\n* line 307: agents number -> number of agents\n* line 314: basically -> mostly\n* Figure 3: temporature -> temperature\n* line 372: conlcusion -> Conclusion\n* line 373: lack of -> lacks\n* line 374: from theories -> using theory\n* line 375: exponentially -> exponential\n* line 379: exponentially -> exponential\n* line 381: achieve -> achieves\n* line 381: Some -> Ablation studies\n* line 382: also made -> conducted\n\n[1]. Kostrikov et. al., Offline Reinforcement Learning with Implicit Q-Learning, ICLR 2022.  \n[2]. Scott Fujimoto, Shixiang Shane Gu, A Minimalist Approach to Offline Reinforcement Learning, NeurIPS 2021.  \n[3]. Peng et. al., Advantage Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, arXiv 2020.\n\nQUESTIONS:\n* What is the formal problem definition of offline RL? What is a behavior policy? What is distribution shift?\n* What is the interpretation of Figure 3 (b)? How does an increase in the size of dataset lead to a decrease in agent performance? How does CFCQL scale for larger data samples?\n* Can you please explain the reasoning behind off-policy/online RL methods being relevant baselines? Can CFCQL be compared to IQL, TD3-BC or AWR even if on small toy tasks?\n* How does CFCQL address distribution shift? How does CFCQL generalize to new task? Can CFCQL be adapted to new multi-agent tasks or transferred to different kinds of agents following pretraining?\n\nLIMITATIONS:\nAuthors have discussed limitations and future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel offline multi-agent reinforcement learning algorithm called Counterfactual Conservative Q-Learning (CFCQL) to address the overestimation issue and achieve team coordination at the same time. The algorithm calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. The paper compares CFCQL and MACQL theoretically and shows that CFCQL is advantageous to MACQL on the performance bounds and safe policy improvement guarantee as the agent number is large. The paper conducts experiments on commonly used multi-agent environments to demonstrate that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.\n\nSTRENGTHS:\n- Theoretical comparison of the proposed algorithm with existing methods to show its advantages in terms of performance bounds and safe policy improvement guarantee.\n- The paper provides a detailed explanation of the proposed algorithm and its counterfactual approach to conservative value estimation.\n- Conducting experiments on multiple environments to demonstrate the effectiveness of the proposed algorithm.\n\nWEAKNESSES:\n- The paper seems to be a simple combination of CQL and MARL.\n- Although the paper shows that the proposed algorithm is theoretically superior to existing methods when the agent number is large, it does not provide empirical evidence to support this claim.\n\ntypos\n- Section \"conclusion\" -> \"Conclusion\"\n\nQUESTIONS:\n- How to get $a_i$ in eq.7? Or what's the range of the summation?\n- To my understanding, does CFCQL update $Q$ for each agent's action iteratively instead of jointly in MACQL? In this case, does the order of the updates matter? And why sample only OOD actions from one agent instead of two or more if two agents' action are highly related?\n\nLIMITATIONS:\nThe paper does not explicitly mention any limitations of the proposed method. The paper has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper addresses challenges in Offline Multi-Agent Reinforcement Learning (MARL), which suffers from severe distribution shift issues and high dimensionality. To overcome these problems, the authors propose a novel MARL algorithm, CounterFactual Conservative Q-Learning (CFCQL). Unlike conventional methods that treat all agents as a single high-dimensional entity, CFCQL separately computes conservative regularization for each agent in a counterfactual manner and linearly combines them for an overall conservative value estimation. The authors demonstrate that CFCQL maintains the underestimation property and performance guarantee similar to single-agent methods but offers improved regularization and safe policy improvement bounds that are independent of the agent number. This method is thus advantageous, especially with large agent counts. Experimental validation on various environments and datasets shows that CFCQL outperforms existing methods significantly.

STRENGTHS:
* The structure of the presentation is generally clear.
* The considered problem is interesting and of importance.

WEAKNESSES:
* The motivation for the algorithm is unclear, e.g. why considering counterfactual is helpful for offline MARL?
* The theoretical analysis is not well elaborated in context. E.g. the implications of theorems can be better elaborated.
* Some terms are not clearly defined, e.g. “PI”, “e_i” in Eq.(8). 
* How algorithm 1  works with continuous or discrete action space is not clear.

QUESTIONS:
See weakness above.

LIMITATIONS:
* The presentation needs to be improved in terms of clarity, and rigor.

* The method is still employing centralized training, which limits its scalability.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23