SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiangsen Wang $^{1*}$  Haoran Xu $^{2*}$  Yinan Zheng $^{3}$  Xianyuan Zhan $^{3,4\dagger}$
<sup>1</sup> Beijing Jiaotong University <sup>2</sup> UT Austin <sup>3</sup> Tsinghua University
$^{4}$  Shanghai Artificial Intelligence Laboratory
wangxiangsen@bjtu.edu.cn, haoran.xu@utexas.edu,
zhengyn23@mails.tsinghua.edu.cn, zhanxianyuan@air.tsinghua.edu.cn
Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline multi-agent RL algorithm with implicit global-to-local value regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks. Our code is available at https://github.com/ZhengYinan-AIR/OMIGA.
Multi-agent reinforcement learning (MARL) is an active research area to tackle many real-world problems that involve multi-agent systems, such as autonomous vehicle coordination[1], network traffic routing [2], and multi-player strategy games [3]. Although MARL has made some impressive progress in recent years, most of its successes are restricted to simulation environments. In most real-world applications, building high-fidelity simulators can be rather costly or even infeasible, and online interaction with the real system during policy learning is also expensive or risky. Inspired by the recently emerged offline RL methods [4, 5, 6, 7, 8, 9, 10], equipping MARL with offline learning capability has become an attractive direction to tackle real-world multi-agent tasks.
Offline RL focuses on learning optimal policies with pre-collected offline datasets with no further environmental interaction. Under the offline setting, evaluating value function on out-of-distribution (OOD) samples can cause extrapolation error accumulation on Q-values, leading to erroneously overestimated values and misleading policy learning [4, 5]. This issue, which is also called distributional shift [5], is one of the core challenges in offline RL. Current offline RL methods tackle this issue primarily by incorporating various forms of data-related regularizations to restrict policy learning
*Work done during the internships with Institute for AI Industry Research (AIR), Tsinghua University.
from deviating too much from the behavioral data. However, extending the same offline regularization technique to the multi-agent setting poses some unique challenges. The joint state-action space grows exponentially with the number of agents, making the global-level regularization hard to compute and may also result in very sparse constraints under the huge joint state-action space, especially when the size and coverage of the offline dataset are limited.
To tackle the challenges, some recent works [11, 12, 13] attempt to design offline MARL algorithms heuristically by enforcing local-level regularizations. By using the Centralized Training with Decentralized Execution (CTDE) framework [14] to decompose the global value function into a combination of local value functions, these works turn the original offline multi-agent RL problem into the offline single-agent RL problem and apply policy or value regularizations at the local level. While straightforward and easy to implement, simply enforcing local-level regularization cannot guarantee the induced regularization at the global-level still remains valid, which can potentially lead to over-conservative policy learning. Existing approaches offer no guarantee whether the optimized local policies are jointly optimal under a given value decomposition scheme. Furthermore, because the offline regularization of most existing methods is completely imposed from the local-level without considering the global information, these methods fail to capture coordinated agent behaviors and credit assignment in the multi-agent system.
In this paper, we aim to give a principled approach to the offline MARL problem. We start by imposing global-level regularizations to form a behavior-regularized MDP. We then find that using some particular regularization choice in this MDP will have closed-form solutions, which naturally turn the global-level regularizations into local-level ones, under some mild assumptions. Finally, we show that the local-level regularization can be derived in an implicit way by using only samples from the offline datasets, without knowing the behavior policy or approximating the regularization term. We dub this new algorithm as offline multi-agent RL algorithm with implicit global-to-local value regularization (OMIGA). In OMIGA, the local-level value regularization is rigorously derived from the global regularization under value decomposition, thus also capturing global information and the impact of multi-agent credit assignment. Under this design, we can also guarantee that the learned local policies are jointly optimal at the global level. Also, OMIGA enables complete in-sample learning without querying OOD action samples, which enjoys better stability during policy learning.
We evaluate our method using various types of offline datasets on both multi-agent MuJoCo [15] and StarCraft Multi-Agent Challenge (SMAC) tasks [16]. Under all settings, OMIGA achieves better performance and enjoys faster convergence compared with other strong baselines.
Offline reinforcement learning. Due to the absence of online data collection, offline policy learning suffers from severe distributional shift and exploitation error accumulation issues when evaluating the value function on OOD actions [4, 5]. As a result, existing offline RL methods adopt various approaches to learn policies pessimistically and regularize policy learning close to data distribution. Policy constraint methods [4, 5, 9, 17, 18, 19] add explicit or implicit policy constraints to regularize the policy from deviating too much from behavioral data. Value regularization methods [20, 21, 22, 23] learn conservatively estimated value functions on OOD data. Uncertainty-based methods add uncertainty penalization terms on value functions [24, 25] or rewards [26, 27, 28] to enable pessimistic policy learning. Recently, in-sample learning methods [7, 8, 10, 29] provide a new direction to avoid the distributional shift by learning value functions and policies completely within data. These methods eliminate the involvement of policy-generated OOD samples during policy evaluation, thus enjoying superior learning stability. Our method embeds multi-agent value decomposition within the in-sample learning paradigm, which fully exploits the multi-agent problem structure for offline policy optimization.
Multi-agent reinforcement learning. The complexity of multi-agent RL problems typically arises from the huge joint action space [30]. In recent years, the CTDE framework [31, 32] has been proposed to decouple agents' learning and execution phases to handle the exploding action space issue. Under the CTDE framework, agents are trained in a centralized manner with global information and make decisions with learned local policies during execution. Some representative works are the value decomposition methods [33, 34, 35, 36], which decompose the global Q-value function into a set of local Q-value functions for scalable multi-agent policy optimization.
There have been some recent attempts to design MARL algorithms for the offline setting, which can be broadly categorized as RL-based [11, 12, 13, 37] and goal-conditioned supervised learning (GCSL) based methods [38, 39]. Existing RL-based methods are typically built upon the CTDE framework and perform offline policy regularization at the local level. For example, ICQ [11] uses importance sampling to implicitly constrain local policy learning on OOD samples. OMAR [12] adopts CQL [20] to learn local Q-value functions and adds zeroth-order optimization to avoid bad local optima. In these methods, the multi-agent modeling and offline RL ingredients are fragmented, which cannot guarantee that the local-level regularizations still remain valid at the global level, and also ignores the cooperative behavior among agents. Moreover, there is no consideration of the value decomposition's influence on the optimal local policies learned with offline regularizations. On the other hand, GCSL-based methods [38, 39] leverage the sequential data modeling capability of transformer architecture to solve offline MARL tasks. However, recent studies also show that GCSL-based methods can be over-conservative [8] and cannot guarantee optimality under stochastic environments [40, 41].
In this work, our proposed OMIGA is an RL-based method, which uses an implicit global-to-local value regularization scheme to tackle the aforementioned limitations, and offers an elegant solution to marry multi-agent modeling and offline learning.
We focus on the fully cooperative multi-agent tasks, which can be modeled as a multi-agent Partially Observable Markov Decision Process (POMDP) [42], defined by a tuple  $G = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, r, \mathcal{Z}, \mathcal{O}, n, \gamma \rangle$ .  $s \in S$  denotes the true state of the environment.  $\mathcal{A}$  is the action set for each of the  $n$  agents. At each step, an agent  $i \in \{1, 2, \ldots, n\}$  chooses an action  $a_i \in \mathcal{A}$ , forming a joint action  $\boldsymbol{a} = (a_1, a_2, \ldots, a_n) \in \mathcal{A}^n$ .  $P(\mathbf{s}'|\mathbf{s}, \boldsymbol{a}) : \mathcal{S} \times \mathcal{A}^n \times \mathcal{S} \rightarrow [0, 1]$  is the transition dynamics to the next state  $s'$ .  $\gamma \in [0, 1)$  is a discount factor. In the partial observable environment, each agent receives an observation  $o_i \in \mathcal{O}$  at each step based on an observation function  $\mathcal{Z}(\mathbf{s}, i) : \mathcal{S} \times N \rightarrow \mathcal{O}$ , and we denote  $\boldsymbol{o} = (o_1, o_2, \ldots, o_n)$ . All agents share the same global reward function  $r(\boldsymbol{o}, \boldsymbol{a}) : \mathcal{O} \times \mathcal{A}^n \rightarrow \mathbb{R}$ . In cooperative MARL, all agents aim to learn a set of policies  $\pi_{tot} = \{\pi_1, \dots, \pi_n\}$  that jointly maximize the expected discounted returns  $\mathbb{E}_{\boldsymbol{a} \in \pi_{tot}, \boldsymbol{o} \in \mathcal{O}} [\sum_{t=0}^{\infty} \gamma^t r(\boldsymbol{o}_t, \boldsymbol{a}_t)]$ . Under the offline setting, a pre-collected dataset  $D$  is obtained by sampling with the behavior policy  $\mu_{tot} = \{\mu_1, \dots, \mu_n\}$  and the policy learning is conducted entirely with the data samples in  $D$  without any environment interactions.
3.2 CTDE Framework and Value Decomposition
In MARL, the joint action space increases exponentially with the increase in the number of agents. Therefore, it is difficult to directly query an optimal joint action from the global Q-value function  $Q_{tot}(o, a)$ , and the global Q-value function could be negatively affected by the suboptimality of individual agents. To address these problems, the Centralized Training with Decentralized Execution (CTDE) framework [14, 31, 32] is proposed. In the training phase, agents can access the full environment information and share each other's experiences. In the execution phase, each agent chooses actions only according to its individual observation  $o_i$ . Through the CTDE framework, optimization at the individual level results in the optimization of the joint action space, which avoids the aforementioned problems. Value decomposition methods [33, 34, 35, 36] are popular solutions under the CTDE framework to achieve the decomposition of the joint action space. Value decomposition is also used in recent offline MARL methods [11, 12], which decomposes the global Q-value function  $Q_{tot}(o, a)$  into a linear combination of local Q-value functions  $Q_i(o_i, a_i)$ :
$$
Q _ {t o t} (\pmb {o}, \pmb {a}) = \sum_ {i} w _ {i} (\pmb {o}) Q _ {i} (o _ {i}, a _ {i}) + b (\pmb {o}),
$$
$$
w _ {i} \geq 0, \forall i = 1 \dots , n \tag {1}
$$
where  $w_{i}(\pmb{o})$  and  $b(\pmb{o})$  capture the weights and offset on local Q-value functions.
3.3 Behavior-Regularized MDP in Offline RL
To avoid the distributional shift in offline RL, existing approaches typically incorporate various data-related regularizations on rewards, value functions, or the policy optimization objective. In our work, we are specifically interested in the treatment that adds a behavior regularizer to the rewards  $r(s,a)$  to penalize OOD samples [10]. This framework leads to a special behavior-regularized MDP, which optimizes a policy with the following objective:
$$
\max  _ {\pi} \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}\right) - \alpha f \left(\pi \left(a _ {t} \mid s _ {t}\right), \mu \left(a _ {t} \mid s _ {t}\right)\right)\right) \right], \tag {2}
$$
where  $\alpha$  is a scale parameter, and  $f(\cdot ,\cdot)$  is the function that captures the divergence between  $\pi$  and  $\mu$ , which is similar to the entropy regularization in SAC [43]. In the above objective, we regularize the deviation between the learned policy  $\pi$  and behavior policy  $\mu$ , so as to avoid the distributional shift issue. This behavior-regularized MDP corresponds to the following modified policy evaluation operator  $\mathcal{T}_f$ :
$$
\left(\mathcal {T} _ {f} ^ {\pi}\right) Q (s, a) := r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} | s, a} \left[ V \left(s ^ {\prime}\right) \right]
$$
$$
\left(\mathcal {T} _ {f} ^ {\pi}\right) V (s) := \mathbb {E} _ {a \sim \pi} \left[ r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} | s, a} \left[ V \left(s ^ {\prime}\right) \right] \right],
$$
$$
V (s) = \mathbb {E} _ {a \sim \pi} \left[ Q (s, a) - \alpha f \left(\pi \left(a _ {t} | s _ {t}\right), \mu \left(a _ {t} | s _ {t}\right)\right) \right].
$$
In the next section, we will derive our proposed method OMIGA based on the behavior-regularized framework, and demonstrate the benefits and desired properties of this framework for the multi-agent setting.
In this section, we formally present our implicit global-to-local value regularization approach OMIGA for offline MARL and explain how it can be integrated into effective offline learning. We begin with the multi-agent POMDP with a reverse KL global value regularization added to the rewards. We then show how we can make an equivalent reformulation that naturally converts the global-level regularization into local-level value regularizations. Lastly, we derive a practical algorithm that implicitly enables local-level value regularizations via in-sample learning and provide a thorough discussion of it.
4.1 Multi-Agent POMDP with Global Value Regularization
When extending Eq.(2) to multi-agent POMDP, we can get the following learning objective:
$$
\max _ {\pi_ {t o t}} \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(\boldsymbol {o} _ {t}, \boldsymbol {a} _ {t}\right) - \alpha f \left(\pi_ {t o t} \left(\boldsymbol {a} _ {t} | \boldsymbol {o} _ {t}\right), \mu_ {t o t} \left(\boldsymbol {a} _ {t} | \boldsymbol {o} _ {t}\right)\right)\right) \right]
$$
However, unlike the single-agent setting, it is difficult to directly compute the regularization term between the global policy  $\pi_{tot}$  and the global behavior policy  $\mu_{tot}$  due to the huge state-action space in multi-agent RL. Can we choose a function  $f$  to allow certain forms of policy decomposition so as to simplify the calculations?
In this work, we find that choosing the regularization function  $f$  to be the reverse KL divergence, which means  $f(\pi_{tot},\mu_{tot}) = \log (\pi_{tot} / \mu_{tot})$ , leads to natural decomposition of the global regularization into the summation of a set of local regularizations, by only assuming the factorization structure of the behavior policy (i.e.,  $\mu_{tot}(a|o) = \prod_{i = 1}^{n}\mu_{i}(a_{i}|o_{i})$ ). Note that using reverse KL divergence as the behavior constraint has been widely studied in both online and offline single-agent RL literature [43, 17], and has been shown to produce the nice mode-seeking behavior to avoid OOD actions in the offline setting [18]. Plug in the specific form of  $f$ , and we obtain the following global policy evaluation operator:
$$
\left(\mathcal {T} _ {f} ^ {\pi_ {t o t}}\right) Q _ {t o t} (\boldsymbol {o}, \boldsymbol {a}) := r (\boldsymbol {o}, \boldsymbol {a}) + \gamma \mathbb {E} _ {\boldsymbol {o} ^ {\prime} | \boldsymbol {o}, \boldsymbol {a}} \left[ V _ {t o t} \left(\boldsymbol {o} ^ {\prime}\right) \right] \tag {3}
$$
$$
\left(\mathcal {T} _ {f} ^ {\pi_ {t o t}}\right) V _ {t o t} (\boldsymbol {o}) := \mathbb {E} _ {\boldsymbol {a} \sim \pi_ {t o t}} \left[ r (\boldsymbol {o}, \boldsymbol {a}) + \gamma \mathbb {E} _ {\boldsymbol {o} ^ {\prime} | \boldsymbol {o}, \boldsymbol {a}} \left[ V _ {t o t} \left(\boldsymbol {o} ^ {\prime}\right) \right] \right], \tag {4}
$$
$$
V _ {t o t} (\pmb {o}) = \mathbb {E} _ {\pmb {a} \sim \pi_ {t o t}} \left[ Q _ {t o t} (\pmb {o}, \pmb {a}) - \alpha \log \left(\frac {\pi_ {t o t} (\pmb {a} | \pmb {o})}{\mu_ {t o t} (\pmb {a} | \pmb {o})}\right) \right] \tag {5}
$$
Theorem 4.1. Define  $\mathcal{T}_f^*$  the case where the global policy in  $\mathcal{T}_f^{\pi_{tot}}$  is the optimal policy  $\pi_{tot}^{*}$ , then  $\mathcal{T}_f^*$  is a  $\gamma$ -contraction.
The proof is in Appendix A. This theorem indicates global Q-value will converge to the Q-value under the optimal policy  $\pi_{tot}^{*}$  when applying the fixed-point iteration with the policy evaluation operator.
We now aim to analyze and derive the closed-form solution of the optimal value functions  $Q_{tot}^{*}$  and  $V_{tot}^{*}$ . According to the Karush-Kuhn-Tucker (KKT) conditions where the derivative of a Lagrangian objective function with respect to the global policy is zero at the optimal solution, we have the following proposition:
Proposition 4.2. For a behavior-regularized multi-agent POMDP with  $f(\pi_{tot}, \mu_{tot}) = \log (\pi_{tot} / \mu_{tot})$ , the optimal global policy  $\pi_{tot}^{*}$  and its optimal value functions  $Q_{tot}^{*}$  and  $V_{tot}^{*}$  satisfy the following optimality condition:
$$
Q _ {t o t} ^ {*} (\boldsymbol {o}, \boldsymbol {a}) = r (\boldsymbol {o}, \boldsymbol {a}) + \gamma \mathbb {E} _ {\boldsymbol {o} ^ {\prime} | \boldsymbol {o}, \boldsymbol {a}} \left[ V _ {t o t} ^ {*} \left(\boldsymbol {o} ^ {\prime}\right) \right]
$$
$$
V _ {t o t} ^ {*} (\boldsymbol {o}) = u ^ {*} (\boldsymbol {o}) + \alpha \tag {6}
$$
$$
\pi_ {t o t} ^ {*} (\pmb {a} | \pmb {o}) = \mu_ {t o t} (\pmb {a} | \pmb {o}) \cdot \exp \left(\frac {Q _ {t o t} ^ {*} (\pmb {o} , \pmb {a}) - u ^ {*} (\pmb {o})}{\alpha} - 1\right)
$$
where  $u(\pmb{o})$  is a normalization term and has a optimal value  $u^{*}$  that makes the corresponding optimal policy  $\pi_{tot}^{*}$  satisfy  $\sum_{\pmb{a} \in \mathcal{A}^n} \pi_{tot}^{*}(\pmb{a}|\pmb{o}) = 1$ .
The proof is in Appendix A. Note that Proposition 4.2 can be further simplified. Since  $V_{tot}^{*}(\boldsymbol{o}) = u^{*}(\boldsymbol{o}) + \alpha$ ,  $u^{*}$  and  $V_{tot}^{*}$  can be converted to each other without any approximation. For the global policy, replacing  $u^{*}(\boldsymbol{o})$  with  $V_{tot}^{*}(\boldsymbol{o}) - \alpha$ , we get the following formulation:
$$
\pi_ {t o t} ^ {*} (\boldsymbol {a} | \boldsymbol {o}) = \mu_ {t o t} (\boldsymbol {a} | \boldsymbol {o}) \cdot \exp \left(\frac {Q _ {t o t} ^ {*} (\boldsymbol {o} , \boldsymbol {a}) - V _ {t o t} ^ {*} (\boldsymbol {o})}{\alpha}\right) \tag {7}
$$
Now we have the relationship among the optimal global policy  $\pi_{tot}^{*}$ , behavior policy  $\mu_{tot}$ , Q-value function  $Q_{tot}^{*}$  and state-value function  $V_{tot}^{*}$ . Under the multi-agent setting, due to the exponential growth of the joint state-action space with the number of agents, these global values are hard to be evaluated. We now show that we can resort to the value decomposition strategy to further derive a tractable relationship between local policies and value functions.
4.2 Global-to-Local Value and Policy Decomposition
In this work, we introduce the following value decomposition scheme for both the global Q-value function and the state-value function:
$$
Q _ {t o t} (\boldsymbol {o}, \boldsymbol {a}) = \sum_ {i} w _ {i} (\boldsymbol {o}) Q _ {i} \left(o _ {i}, a _ {i}\right) + b (\boldsymbol {o})
$$
$$
V _ {t o t} (\boldsymbol {o}) = \sum_ {i} w _ {i} (\boldsymbol {o}) V _ {i} \left(o _ {i}\right) + b (\boldsymbol {o}) \tag {8}
$$
$$
w _ {i} \geq 0, \forall i = 1 \dots , n
$$
Compared with existing offline MARL methods [11, 12, 13] that only decompose the global Q-value function  $Q_{tot}$ , we additionally decompose the global state-value function  $V_{tot}$ . The decomposition of  $Q_{tot}$  and  $V_{tot}$  share a common weight function  $w_{i}(\boldsymbol{o})$ , since the credit assignment on  $Q$  and  $V$  should be related. It should also be noted that  $V_{tot}$  is free of the joint action space and thus not affected by OOD actions under offline learning.
If we incorporate the value decomposition scheme Eq. (8) into the optimal global policy  $\pi_{tot}^{*}$  in Eq. (7) and utilize the property of the exponential function, we can naturally decompose the optimal global policy  $\pi_{tot}^{*}$  into a combination of optimal local policies  $\pi_{i}^{*}$ .
Proposition 4.3. Under the value decomposition scheme specified in Eq. (8) and assume the global behavior policy is decomposable (i.e.,  $\mu_{tot}(\pmb{a}|\pmb{o}) = \prod_{i=1}^{n}\mu_i(a_i|o_i)$ ), we have  $\pi_{tot}^*(\pmb{a}|\pmb{o}) = \prod_{i=1}^{n}\pi_i^*(a_i|o_i)$ , where  $\pi_i^*$  is defined as:
$$
\pi_ {i} ^ {*} \left(a _ {i} \mid o _ {i}\right) = \mu_ {i} \left(a _ {i} \mid o _ {i}\right) \cdot \exp \left(\frac {w _ {i} (\boldsymbol {o})}{\alpha} \left(Q _ {i} ^ {*} \left(o _ {i} , a _ {i}\right) - V _ {i} ^ {*} \left(o _ {i}\right)\right)\right) \tag {9}
$$
The result immediately follows by observing that:
$$
\begin{array}{l} \pi_ {t o t} ^ {*} (\pmb {a} | \pmb {o}) = \mu_ {t o t} (\pmb {a} | \pmb {o}) \cdot \exp \left(\frac {\sum_ {i} w _ {i} (\pmb {o}) (Q _ {i} ^ {*} (o _ {i} , a _ {i}) - V _ {i} ^ {*} (o _ {i}))}{\alpha}\right) \\ = \prod_ {i = 1} ^ {n} \mu_ {i} \left(a _ {i} \mid o _ {i}\right) \cdot \exp \left(\frac {w _ {i} (\boldsymbol {o})}{\alpha} \left(Q _ {i} ^ {*} \left(o _ {i} , a _ {i}\right) - V _ {i} ^ {*} \left(o _ {i}\right)\right)\right) = \prod_ {i = 1} ^ {n} \pi_ {i} ^ {*} \left(a _ {i} \mid o _ {i}\right) \\ \end{array}
$$
$\pi_i^*$  can be perceived as the optimal local policy under the behavior regularization, since the RHS of Eq. (9) only depends on the optimal local value functions  $Q_{i}^{*},V_{i}^{*}$ , and the local behavior policy  $\mu_{i}$  of each agent  $i$ . This policy decomposition structure has a number of attractive characteristics. First, this design naturally bridges global value regularization and value decomposition. Second,  $w_{i}(\pmb{o})$  explicitly appears in the optimal local policy  $\pi_i^*$ , which is calculated from global observation  $\pmb{o}$ . This makes the local policy estimizable with global information and consistent with the credit assignment in the multi-agent system.
4.3 Equivalent Implicit Local Value Regularizations