SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianzhun Shao\*, Yun Qu\* Chen Chen, Hongchang Zhang, Xiangyang Ji
Tsinghua University, Beijing, China
{sjz18, qy22, hc-zhang19}@mails.tsinghua.edu.cn
cclvr@163.com
xyji@tsinghua.edu.cn
Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. To mitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.
Online Reinforcement Learning (Online RL) needs frequently deploying untested policies to environment for data collection and policy optimization, making it dangerous and inefficient to apply in the real-world scenarios (e.g. autonomous vehicle teams). While, Offline Reinforcement Learning (Offline RL) aims to learn policies from a fixed dataset rather than from interacting with the environment, and therefore is suitable for the real applications with highly safety requirements or without efficient simulators [25].
Directly applying off-policy RL to the offline setting may fail due to overestimation [13, 24]. Existing works usually tackle this problem by pessimism. They either utilize behavior regularization to constrain the learning policy close to the behavior policy induced by the dataset [49, 20, 11], or conduct conservative (pessimistic) value iteration to mitigate unexpected overestimation [18, 21]. It has been demonstrated both theoretically and empirically that these methods can achieve comparable performance to their online counterparts under some conditions [21].
Though cooperative Multi-agent Reinforcement Learning (MARL) has gained extraordinary success in various multiplayer video games such as Dota [4], StarCraft [40] and soccer [23], applying current MARL in real scenarios is still challenging due to the same safety and efficiency concerns in single-agent setting, then it is worth conducting investigation for offline RL in multi-agent setting. Compared with single-agent setting, offline RL in multi-agent setting has its own difficulty. On the one hand, the action space explodes exponentially as the agent number increases, then an arbitrary joint action is more likely to be an OOD action given the fixed dataset size, making the OOD phenomenon more severe. This will further exacerbate the issues of extrapolation error and overestimation in the policy evaluation process and thus induce an unexpected or even disastrous final policy [52]. On the other hand, as the core of MARL, the agents need to consider not only their own actions but also other agents' actions as well as contributions to the global return in order to achieve an overall high performance, which undoubtedly increases the difficulty for theoretical analysis. Besides, instead of just guaranteeing single policy improvement, the bounded team performance is also a key concern to offline MARL.
There exist few works to combine MARL with offline RL. Pan et al. [34] uses Independent Learning [44] as a simple solution, i.e., each agent regards others as part of the environment and independently performs offline RL learning. Although mitigating the joint action OOD issue through decoupling the agents's learning completely, it essentially still adopts a single-agent paradigm to learn and thus cannot enjoy the recent progress on MARL that a centralized value function can empower better team coordination [37, 27]. To utilize the advantage of centralized training with decentralized execution (CTDE), Yang et al. [52] applies implicit constraint approach on the value decomposition network [43], which alleviates the extrapolation error and gains some performance improvement empirically. Although it proposes some theoretical analysis for the convergence property of value function, whether the team performance can be bounded from below as agents number increases remains still unknown.
In this paper, we introduce a novel offline MARL algorithm called Counterfactual Conservative Q-Learning (CFCQL) and aim to address the overestimation issue rooted from joint action OOD phenomenon. It adopts CTDE paradigm to realize team coordination, and incorporates the state-of-the-art offline RL algorithm CQL to conduct conservative value estimation. CQL is preferred due to its theoretical guarantee and flexibility of implementation. One direct treatment is to regard all the agents as a single one and conduct standard CQL learning on the joint policy space which we call MACQL. However, too much conservatism will be generated in this way since the induced penalty can be exponentially large in the joint policy space. Instead, CFCQL separately regularizes each agent in a counterfactual way to avoid too much conservatism. Specifically, each agent separately contributes CQL regularization for the global Q value and then a weighted average is used as an overall regularization. When calculating agent  $i$ 's regularization term, rather than sampling OOD actions from the joint action space as MACQL does, we only sample OOD actions for agent  $i$  and leave other agents' actions sampled in the dataset. We prove that CFCQL enjoys underestimation property as CQL does and the safe policy improvement guarantee independent of the agents number  $n$ , which is advantageous to MACQL especially under the situation with a large  $n$ .
We conduct experiments on 1 man-made environment Equal Line, and 3 commonly used multi-agent environments: StarCraft II [40], Multi-agent Particle Environment [27], and Multi-agent MuJoCo [35], including both discrete and continuous action space setting. With datasets collected by Pan et al. [34] and ourselves, our method outperforms existing methods in most settings and even with a large margin on some of them.
We summarize our contributions as follows: (1) we propose a novel offline MARL method CFCQL based on CTDE paradigm to address the overestimation issue and achieve team coordination at the same time. (2) We theoretically compare CFCQL and MACQL to show that CFCQL is advantageous to MACQL on the performance bounds and safe policy improvement guarantee as agent number is large. (3) In hard multi-agent offline tasks with both discrete and continuous action space, our method shows superior performance to the state-of-the-art.
Offline RL. Standard RL algorithms are especially prone to fail due to erroneous value overestimation induced by the distributional shift between the dataset and the learning policy. Theoretically, it is
proved that pessimism can alleviate overestimation effectively and achieve good performance even with non-perfect data coverage [5, 15, 26, 22, 39, 51, 55, 6]. In the algorithmic line, there are broadly two categories: uncertainty based ones and behavior regularization based ones. Uncertainty based approaches attempt to estimate the epistemic uncertainty of Q-values or dynamics, and then utilize this uncertainty to pessimistically estimating Q in a model-free manner [2, 50, 3], or conduct learning on the pessimistic dynamic model in a model-based manner [54, 16]. Behavior regularization based algorithms constrain the learned policy to lie close to the behavior policy in either explicit or implicit ways [20, 13, 49, 18, 21, 11], and is advantageous over uncertainty based methods in computation efficiency and memory consumption. Among these class of algorithms, CQL[21] is preferred due to its superior empirical performance and flexibility of implementation.
MARL. A popular paradigm for multi-agent reinforcement learning is centralized training with decentralized execution (CTDE) [37, 27]. Centralized training inherits the idea of joint action learning [7], empowering the agents with better team coordination. And decentralized execution enjoys the deploying flexibility of independent learning [44]. In CTDE, some value-based works concentrate on decomposing the single team reward to all agents by value function factorization [43, 37], based on which they further derive and extend the Individual-Global-Max (IGM) principle for policy optimality analysis [42, 47, 38, 46]. Another group of works focus on the actor-critic framework, using a centralized critic to guide each agent's policy update [27, 9]. Some variants of PPO [41], including IPPO [8], MAPPO [53], and HAPPO [19] also show great potential in solving complex tasks. All these works use the online learning paradigm, therefore disturbed by extrapolation error when transferred to the offline setting.
Offline MARL. To make MARL applied in the more practical scenario with safety and training efficiency concerns, offline MARL is proposed. Jiang & Lu [14] shows the challenge of applying BCQ [13] to independent learning, and Pan et al. [34] uses zeroth-order optimization for better coordination among agents' policies. Although it empirically shows fast convergence rate, the policies trained by independent learning have no theoretical guarantee for the team performance. With CTDE paradigm, Yang et al. [52] adopts the same treatment as Peng et al. [36] and Nair et al. [30] to avoid sampling new actions from current policy, and Tseng et al. [45] regards the offline MARL as a sequence modeling problem, solving it by supervised learning. As a result, both methods' performance relies heavily on the data quality. In contrast, CFCQL does not require the learning policies stay close to the behavior policy, and therefore performs well on datasets with low quality. Meanwhile, CFCQL is theoretically guaranteed to be safely improved, which ensures its performance on datasets with high quality.
3.1 MARL Symbols
We use a decentralised partially observable Markov decision process (Dec-POMDP) [32]  $G = \langle S, \mathbf{A}, I, P, r, Z, O, n, \gamma \rangle$  to model a fully collaborative multi-agent task with  $n$  agents, where  $s \in S$  is the global state. At time step  $t$ , each agent  $i \in I \equiv \{1, \dots, n\}$  chooses an action  $a^i \in A$ , forming the joint action  $\mathbf{a} \in \mathbf{A} \equiv A^n$ .  $T(s'|s, \mathbf{a}) : S \times \mathbf{A} \times S \to [0,1]$  is the environment's state transition distribution. All agents share the same reward function  $r(s, \mathbf{a}) : S \times \mathbf{A} \to \mathbb{R}$ .  $\gamma \in [0,1)$  is the discount factor. Each agent  $i$  has its local observations  $o^i \in O$  drawn from the observation function  $Z(s,i) : S \times I \to O$  and chooses an action by its policy  $\pi^i(a^i|o^i) : O \to \Delta([0,1]^{|A|})$ . The agents' joint policy  $\pi := \prod_{i=1}^{n} \pi^i$  induces a joint action-value function:  $Q^\pi(s, \mathbf{a}) = \mathbb{E}[R|s, \mathbf{a}]$ , where  $R = \sum_{t=0}^{\infty} \gamma^t r_t$  is the discounted accumulated team reward. We assume  $\forall r, |r| \leq R_{\max}$ . The goal of MARL is to find the optimal joint policy  $\pi^*$  such that  $Q^\pi^*(s, \mathbf{a}) \geq Q^\pi(s, \mathbf{a})$ , for all  $\pi$  and  $(s, \mathbf{a}) \in S \times \mathbf{A}$ . In offline MARL, we need to learn the optimal  $\pi^*$  from a fixed dataset sampled by an unknown behaviour policy  $\beta$ , and we can not deploy any new policy to the environment to get feedback.
3.2 Value Functions in MARL
In MARL settings with discrete action space, we can maintain for each agent a local-observation-based Q function  $Q^{i}(o^{i},a^{i})$ , and define the local policy  $\pi^i \coloneqq \arg \max_a Q^i (o^i,a)$ . To train local policies with single team reward, Rashid et al. [37] proposes QMIX, using a global  $Q^{tot}$  to model
the cumulative team reward. Define the Bellman operator  $\mathcal{T}$  as  $\mathcal{T}Q(s,\pmb{a}) = r + \gamma \max_{\pmb{a}'} Q(s',\pmb{a}')$ . QMIX aims to minimize the temporal difference:
$$
\hat {Q} _ {k + 1} ^ {t o t} \leftarrow \underset {Q} {\arg \min } \mathbb {E} _ {s, \boldsymbol {a}, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q (s, \boldsymbol {a}) - \hat {\mathcal {T}} \hat {Q} _ {k} ^ {t o t} (s, \boldsymbol {a})\right) ^ {2} \right], \tag {1}
$$
where  $\hat{\mathcal{T}}$  is the empirical Bellman operator using samples from the replay buffer  $\mathcal{D}$ , and  $\hat{Q}^{tot}$  is the empirical global Q function, computed from  $s$  and  $Q^i$ s represented by a neural network satisfying  $\partial Q^{tot} / \partial Q^i\geq 0$ . With such restriction, QMIX can ensure the Individual-Global-Max principle that  $\arg \max_{\mathbf{a}}Q^{tot}(s,\mathbf{a}) = (\arg \max_{a^1}Q^1 (o^1,a^1),\dots,\arg \max_{a^n}Q^n (o^n,a^n))$ . For continuous action space, a neural network is used to represent the local policy  $\pi^i$  for each agent. Lowe et al. [27] proposes MADDPG, maintaining a centralized critic  $Q^{tot}$  to directly guide the local policy update by gradient descent. And the update rule of  $Q^{tot}$  is similar to Eq. 1, just replacing  $\mathcal{T}$  with  $\mathcal{T}^\pi$ .  $\mathcal{T}^\pi Q = r + \gamma P^\pi Q$ , where  $P^\pi Q(s,\pmb {a}) = \mathbb{E}_{s'\sim T(\cdot |s,\pmb {a}),\pmb{a}'\sim \pi (\cdot |s')}[Q(s',\pmb{a}')]$ .
3.3 Conservative Q-Learning
Offline RL is prone to the overestimation of value functions. For an intuitive explanation of the underlying cause, please refer to Appendix E.1. Kumar et al. [21] proposes Conservative Q-Learning (CQL), adding a regularizer to the Q function to address the overestimation issue, which maximizes the Q function of the state-action pairs from the dataset distribution  $\beta$ , while penalizing the Q function sampled from a new distribution  $\mu$  (e.g., current policy  $\pi$ ). The conservative policy evaluation is shown as follows:
$$
\hat {Q} _ {k + 1} \leftarrow \underset {Q} {\arg \min } \alpha \left[ \mathbb {E} _ {s \sim \mathcal {D}, \alpha \sim \mu} [ Q (s, a) ] - \mathbb {E} _ {s \sim \mathcal {D}, \alpha \sim \beta} [ Q (s, a) ] \right] + \frac {1}{2} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \left(Q (s, a) - \hat {\mathcal {T}} ^ {\pi} \hat {Q} _ {k} (s, a)\right) ^ {2} \right]. \tag {2}
$$
As shown in Theorem 3.2 in Kumar et al. [21], with a large enough  $\alpha$ , we can obtain a Q function, whose expectation over actions is lower than that of the real Q, then the overestimation issue can be mitigated.
4 Proposed Method
We first show the overestimation problem of multi-agent value functions in Sec. 4.1, and a direct solution by the naive extension of CQL to multi-agent settings in Sec. 4.2, which we call MACQL. In Sec. 4.3, we propose Counterfactual Conservative Q-Learning (CFCQL) and further demonstrate that it can realize conservatism in a more mild and controllable way which is independent of the number of agents. Then we compare MACQL and CFCQL from the aspect of safe improvement performance. In Sec. 4.4, we present our novel and practical offline RL algorithm.
4.1 Overestimation in Offline MARL
The Q-values in online learning usually face the overestimation problem [1, 33], which becomes more severe in offline settings due to distribution shift and offline policy evaluation [13]. Offline MARL suffers from the same issue even worse since the joint action space explodes exponentially as the agent number increases, then an arbitrary joint action is more likely to be an OOD action given the fixed dataset size. To illustrate this phenomenon, we design a toy Multi-Agent Markov Decision Process (MMDP) as shown in Fig.1(a). There are five agents and three states. All agents are randomly initialized and need learn to take three actions
Figure 1: (a) A decomposable Multi-Agent MDP which urges the agents moves to or stays in  $S2$ . The number of agents is set to 5. (b) The learning curve of the estimated joint state-action value function in the given MMDP. The true value is indicated by the dotted line, which represents the maximum discounted return. (c) The performance curve of the corresponding methods.
(stay, move up, move down) to move to or stay in the state  $S2$ . The dataset consists of 1000 samples with reward larger than  $0.8r_{\mathrm{max}}$ . As shown in Fig. 1(b), if not specially designed for offline setting,
naive application of QMIX leads to exponentially explosion of value estimates and the sub-optimal policy performance, Fig. 1(c).
4.2 Multi-Agent Conservative Q-Learning
When treating multiple agents as a unified single agent and only regarding the joint policy of the  $\beta, \mu, \pi$ , that is  $\beta(a|s) = \prod_{i=1}^{n} \beta^i(a^i|s)$ ,  $\mu(a|s) = \prod_{i=1}^{n} \mu^i(a^i|s)$  and  $\pi(a|s) = \prod_{i=1}^{n} \pi^i(a^i|s)$ , we can extend the single-agent CQL to CTDE multi-agent setting straightforwardly and derive the conservative policy evaluation style directly as follows:
$$
\hat {Q} _ {k + 1} ^ {t o t} \leftarrow \underset {Q} {\arg \min } \alpha \left[ \mathbb {E} _ {s \sim \mathcal {D}, \boldsymbol {a} \sim \boldsymbol {\mu}} [ Q (s, \boldsymbol {a}) ] - \mathbb {E} _ {s \sim \mathcal {D}, \boldsymbol {a} \sim \beta} [ Q (s, \boldsymbol {a}) ] \right] + \hat {\mathcal {E}} _ {\mathcal {D}} (\boldsymbol {\pi}, Q, k), \qquad (3)
$$
where we represent the temporal difference (TD) error concisely as  $\hat{\mathcal{E}}_{\mathcal{D}}(\pi, Q, k) := \frac{1}{2} \mathbb{E}_{s, \boldsymbol{a}, s' \sim \mathcal{D}}[(Q(s, \boldsymbol{a}) - \hat{T}^{\pi} \hat{Q}_k^{tot}(s, \boldsymbol{a}))^2]$ . In the rest of the paper, we mainly focus on the global Q function of a centralized critic and omit the superscript "tot" for ease of expression.
Similar to CQL, MACQL can learn a lower-bounded Q-value with a proper  $\alpha$ , then the overestimation issue in offline MARL can be mitigated. According to Theorem 3.2 in Kumar et al. [21], the degree of pessimism relies heavily on  $D_{CQL}(\pi, \beta)(s) \coloneqq \sum_{a} \pi(a|s)[\frac{\pi(a|s)}{\beta(a|s)} - 1]$ . However, as we will show in Sec. 4.3,  $D_{CQL}$  expands exponentially as the number of agents  $n$  increases, resulting in an over-pessimistic value function and a mediocre policy improvement guarantee for MACQL.
4.3 Counterfactual Conservative Q-Learning
Intuitively, we aim to learn a value function that prevents the overestimation of the policy value, which is expected not too far away from the true value in the meantime. Similar to MACQL as introduced above, we add a regularization in the policy evaluation process, by penalizing the values for OOD actions and rewarding the values for in-distribution actions. However, rather than regarding all the agents as a single one and conducting standard CQL method in the joint policy space, in CFCQL method, each agent separately contributes regularization for the global Q value and then a weighted average is used as an overall regularization. When calculating agent  $i$ 's regularization term, instead of sampling OOD actions from the joint action space as MACQL does, we only sample OOD actions for agent  $i$  and leave other agents' actions sampled in the dataset. Specifically, using  $a^{-i}$  to denote the joint actions other than agent  $i$ . Eq. 4 is our proposed policy evaluation iteration:
$$
\hat {Q} _ {k + 1} \leftarrow \underset {Q} {\arg \min } \alpha \left[ \sum_ {i = 1} ^ {n} \lambda_ {i} \mathbb {E} _ {s \sim \mathcal {D}, a ^ {i} \sim \mu^ {i}, \boldsymbol {a} ^ {- i} \sim \beta^ {- i}} [ Q (s, \boldsymbol {a}) ] - \mathbb {E} _ {s \sim \mathcal {D}, \boldsymbol {a} \sim \beta} [ Q (s, \boldsymbol {a}) ] \right] + \hat {\mathcal {E}} _ {\mathcal {D}} (\boldsymbol {\pi}, Q, k), \tag {4}
$$
where  $\alpha, \lambda_{i}$  are hyper-parameters, and  $\sum_{i=1}^{n} \lambda_{i} = 1, \lambda_{i} \geq 0$ . We refer to our method as 'counterfactual' because its structure bears resemblance to counterfactual methods in MARL [9]. This involves obtaining each agent's counterfactual baseline by marginalizing out of a single agent's action while keeping the other agents' actions fixed. The intuitive rationale behind employing a counterfactual-like approach is that by individually penalizing each agent's out-of-distribution (OOD) actions while holding the others' actions constant from the datasets, we can effectively mitigate the out-of-distribution problem in offline MARL with reduced pessimism, as illustrated in the rest of this section.
The theoretical analysis is arranged as follows: In Theorem 4.1 we show the new policy evaluation leads to a milder conservatism on value function, which still lower bounds the true value. Then we compare the conservatism degree between CFCQL and MACQL in Theorem 4.2. In Theorem 4.3 and Theorem 4.4 we show the effect of milder conservatism brought by counterfactual treatments on the performance guarantee (All proofs are deferred to Appendix A).
Theorem 4.1 (Equation 4 results in a lower bound of value function). The value of the policy under the  $Q$  function from Equation 4,  $\hat{V}^{\pi}(s) = \mathbb{E}_{\pi(\boldsymbol{a}|s)}[\hat{Q}^{\pi}(s,\boldsymbol{a})]$ , lower-bounds the true value of the policy obtained via exact policy evaluation,  $V^{\pi}(s) = \mathbb{E}_{\pi(\boldsymbol{a}|s)}[Q^{\pi}(s,\boldsymbol{a})]$ , when  $\boldsymbol{\mu} = \boldsymbol{\pi}$ , according to:  $\forall s \in$
$$
\mathcal {D}, \hat {V} ^ {\pi} (s) \leq V ^ {\pi} (s) - \alpha \left[ (I - \gamma P ^ {\pi}) ^ {- 1} \mathbb {E} _ {\pi} \left[ \sum_ {i = 1} ^ {n} \lambda_ {i} \frac {\pi^ {i}}{\beta^ {i}} - 1 \right] \right] (s) + \left[ (I - \gamma P ^ {\pi}) ^ {- 1} \frac {C _ {r , T , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right].
$$
Define  $D_{CQL}^{CF}(\pmb {\pi},\pmb {\beta})(s)\coloneqq \sum_{\pmb{a}}\pmb {\pi}(\pmb {a}|s)[\sum_{i = 1}^{n}\lambda_{i}\frac{\pi^{i}(a^{i}|s)}{\beta^{i}(a^{i}|s)} -1]$ . If  $\alpha >\frac{C_{r,T,\delta}R_{\max}}{1 - \gamma}\cdot \max_{s\in \mathcal{D}}\frac{1}{\sqrt{|\mathcal{D}(s)|}}$
$D_{CQL}^{CF}(\pmb {\pi},\pmb {\beta})(s)^{-1},\forall s\in \mathcal{D},\hat{V}^{\pmb{\pi}}(s)\leq V^{\pmb{\pi}}(s),$  with probability  $\geq 1 - \delta$  . When  $\mathcal{T}^{\pi} = \hat{\mathcal{T}}^{\pi}$  , then any  $\alpha >0$  guarantees  $\hat{V}^{\pi}(s)\leq V^{\pi}(s),\forall s\in \mathcal{D}$
Theorem 4.1 mainly differs from Theorem 3.2 in Kumar et al. [21] on the specific form of the divergence:  $D_{CQL}$  and  $D_{CQL}^{CF}$ , both of which determine the degrees of conservatism when  $\pi$  and  $\beta$  are different. Empirically, we find  $D_{CQL}$  generally becomes too large when the number of agents expands, resulting in a low, even negative value function. To measure the scale difference between  $D_{CQL}(\pi, \beta)(s) \coloneqq \prod_i (D_{CQL}(\pi^i, \beta^i)(s) + 1) - 1$  and  $D_{CQL}^{CF}(\pi, \beta)(s) \coloneqq \sum_i \lambda_i D_{CQL}(\pi^i, \beta^i)(s)$ , we have Theorem 4.2:
Theorem 4.2.  $\forall s, \pi, \beta$ , one has  $0 \leq D_{CQL}^{CF}(\pi, \beta)(s) \leq D_{CQL}(\pi, \beta)(s)$ , and the following inequality holds:
$$
\frac {D _ {C Q L} (\boldsymbol {\pi} , \boldsymbol {\beta}) (s)}{D _ {C Q L} ^ {C F} (\boldsymbol {\pi} , \boldsymbol {\beta}) (s)} \geq \exp \left(\sum_ {i = 1, i \neq j} ^ {n} K L \left(\pi^ {i} (s) \mid \mid \beta^ {i} (s)\right)\right), \tag {5}
$$
where  $j = \arg \max_k\mathbb{E}_{\pi^k}\frac{\pi^k(s)}{\beta^k(s)}$  represents the agent whose policy distribution is the most far away from the dataset.