SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daiki E. Matsunaga *
dematsunaga@ai.kaist.ac.kr
Jongmin Lee *
jongmin.lee@berkeley.edu
Jaeseok Yoon
jsyoon@ai.kaist.ac.kr
Stefanos Leonardos
stefanos.leonardos@kcl.ac.uk
Pieter Abbeel
pabbeel@cs.berkeley.edu
Kee-Eung Kim
kekim@kaist.ac.kr
One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. This challenge is amplified in the offline Multi-Agent RL (MARL) setting since the joint action space grows exponentially with the number of agents. To avoid this curse of dimensionality, existing MARL methods adopt either value decomposition methods or fully decentralized training of individual agents. However, even when combined with standard conservatism principles, these methods can still result in the selection of OOD joint actions in offline MARL. To this end, we introduce AlberDICE, an offline MARL algorithm that alternatively performs centralized training of individual agents based on stationary distribution optimization. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time while effectively avoiding OOD joint action selection. Theoretically, we show that the alternating optimization procedure converges to Nash policies. In the experiments, we demonstrate that AlberDICE significantly outperforms baseline algorithms on a standard suite of MARL benchmarks.
Offline Reinforcement Learning (RL) has emerged as a promising paradigm to train RL agents solely from pre-collected datasets [13, 16]. Offline RL aims to address real-world settings in which further interaction with the environment during training is dangerous or prohibitively expensive, e.g., autonomous-car driving, healthcare operations or robotic control tasks [3, 4, 39]. One of the main challenges for successful offline RL is to address the distribution shift that arises from the difference between the policy being learned and the policy used for data collection. Conservatism is a commonly adopted principle to mitigate the distribution shift, which prevents the selection of OOD actions via conservative action-value estimates [11] or direct policy constraints [6].
However, avoiding the selection of OOD actions becomes very challenging in offline Multi-Agent RL (MARL)<sup>2</sup>, as the goal is now to stay close to the states and joint actions in the dataset. This is
2We assume cooperative MARL for this paper in which agents have common rewards.
not trivial since the joint action space scales exponentially with the number of agents, a problem known as the curse of dimensionality. Previous attempts to address these issues include decomposing the joint action-value function under strict assumptions such as the Individual-Global-Max (IGM) principle [30, 33, 36, 40], or decentralized training which ignores the non-stationarity caused by the changing policies of other agents [18, 25, 38]. While effective in avoiding the curse of dimensionality, these assumptions are insufficient in avoiding OOD joint action selection even when applying the conservatism principles.
To illustrate the problem of joint action OOD, consider the XOR game in Figure 1. In this game, two agents need to coordinate to achieve optimal joint actions, here, either  $(A,B)$  or  $(B,A)$ . Despite its simple structure, the co-occurrence of two global optima causes many existing algo
rithms to degenerate in the XOR game [5]. To see this, suppose we have an offline dataset  $D = \{(A,A,0),(A,B,1),(B,A,1)\}$ . In this situation, IGM-based methods [40] represent the joint  $Q(a_{1},a_{2})$  as a combination of individual  $Q_{1}(a_{1})$  and  $Q_{2}(a_{2})$ , where action  $B$  is incentivized over action  $A$  by both agents in the individual  $Q$  functions. As a consequence, IGM-based methods end up selecting  $(B,B)$ , which is the OOD joint action. Similarly,
decentralized training methods [25] also choose the OOD joint action  $(B, B)$ , given that each agent assumes that another agent is fixed with a data policy of selecting  $A$  with probability  $\frac{2}{3}$ . Furthermore, we can see that even behavior-cloning on the expert-only dataset, i.e.,  $D = \{(A, B), (B, A)\}$ , may end up selecting OOD joint actions as well: each individual policy  $\pi_1(a_1)$  and  $\pi_2(a_2)$  will be uniform over the two individual actions, leading to uniform action selection over the entire joint action space; thus, both  $(A, A)$  and  $(B, B)$  can be selected. Consequently, OOD joint actions can be hard to avoid especially in these types of environments with multiple global optima and/or when the offline dataset consists of trajectories generated by a mixture of data collection policies.
Figure 1: XOR Game
Our approach and results To address these challenges, we introduce AlberDICE (ALternate BEst Response Stationary Distribution Correction Estimation), a novel offline MARL algorithm for avoiding OOD actions in the joint action space while circumventing the curse of dimensionality. We start by presenting a coordinate descent-like training procedure where each agent sequentially computes their best response policy while fixing the policies of others. In order to do this in an offline manner, we utilize the linear programming (LP) formulation of RL for optimizing stationary distribution, which has been adapted in offline RL [14] as a stable training procedure where value estimations of OOD actions are eliminated. Furthermore, we introduce a regularization term to the LP objective which matches the stationary distributions of the dataset in the joint action space. This regularization term allows AlberDICE to avoid OOD joint actions as well as the curse of dimensionality without any restrictive assumptions such as factorization of value functions via IGM or fully decentralized training. Overall, our training procedure only requires the mild assumption of Centralized Training and Decentralized Execution (CTDE), a popular paradigm in MARL [17, 28, 30] where we assume access to all global information such as state and joint actions during training while agents act independently during execution.
Theoretically, we show that our regularization term preserves the common reward structure of the underlying task and that the sequence of generated policies converges to a Nash policy (Theorem 4.2). We also conduct extensive experiments to evaluate our approach on a standard suite of MARL environments including the XOR Game, Bridge [5], Multi-Robot Warehouse [27], Google Research Football [12] and SMAC [31], and show that AlberDICE significantly outperforms baselines. To the best of our knowledge, AlberDICE is the first DICE-family algorithm successfully applied to the MARL setting while addressing the problem of OOD joint actions in a principled manner. $^3$
Multi-Agent MDP (MMDP) We consider the fully cooperative MARL setting, which can be formalized as a Multi-Agent Markov Decision Process (MMDP) [24]  $^4$ . An  $N$ -Agent MMDP is defined by a tuple  $G = \langle \mathcal{N},\mathcal{S},\mathcal{A},r,P,p_0,\gamma \rangle$  where  $\mathcal{N} = \{1,2,\dots ,N\}$  is the set of agent indices,  $s\in S$  is the state,  $\mathcal{A} = \mathcal{A}_1\times \dots \times \mathcal{A}_N$  is the joint action space,  $p_0\in \Delta (\mathcal{S})$  is the initial state
$^{3}$ Our code is available at https://github.com/dematsunaga/alberdice
<sup>4</sup>We consider MMDPs rather than Dec-POMDP for simplicity. However, our method can be extended to the Dec-POMDP settings as shown in the experiments. We provide further details in Appendix J.1
distribution, and  $\gamma \in (0,1)$  is the discount factor. At each time step, every agent selects an action  $a_{i}\in \mathcal{A}_{i}$  via a policy  $\pi_i(a_i|s)$  given a state  $s\in S$ , and receives a reward according to a joint (among all agents) reward function  $r:S\times \mathcal{A}\to \mathbb{R}$ . Then, the state is transitioned to the next state  $s^{\prime}$  via the transition probability  $P:S\times \mathcal{A}\rightarrow \Delta (\mathcal{S})$  which depends on the global state  $s\in S$  and the joint action  $\mathbf{a} = \{a_1,\dots ,a_N\} \in \mathcal{A}$  and the process repeats. We also use  $\mathbf{a}_{-i}$  to denote the actions taken by all agents other than  $i$ .
Given a joint policy  $\pi = \{\pi_1, \dots, \pi_N\}$  (or  $\pi = \{\pi_i, \pi_{-i}\}$  when we single out agent  $i \in \mathcal{N}$ ), we have that  $\pi(\mathbf{a}|s) = \pi_1(a_1|s) \cdot \pi_2(a_2|s, a_1) \cdots \pi_N(a_N|s, a_1, \dots, a_{N-1})$ . If the  $\pi_i$ 's can be factorized into a product of individual policies, then  $\pi$  is factorizable, and we write  $\pi(\mathbf{a}|s) = \prod_{j=1}^{N} \pi_j(a_j|s)$ . We denote the set of all policies as  $\Pi$ , and the set of factorizable policies as  $\Pi_f$ . Given a joint policy  $\pi$ , a state value function  $V^{\pi}$  and a action-value function  $Q^{\pi}$  are defined by:
$$
V ^ {\pi} (s) := \mathbb {E} _ {\boldsymbol {\pi}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, \mathbf {a} _ {t}\right) \mid s _ {0} = s \right], \quad Q ^ {\pi} (s, \mathbf {a}) := \mathbb {E} _ {\boldsymbol {\pi}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, \mathbf {a} _ {t}\right) \mid s _ {0} = s, \mathbf {a} _ {0} = \mathbf {a} \right]
$$
The goal is to find a factorizable policy  $\pi \in \Pi_f$  that maximizes the joint rewards so that each agent can act in a decentralized manner during execution. We will consider solving MMDP in terms of optimizing stationary distribution. For a given policy  $\pi$ , its stationary distribution  $d^{\pi}$  is defined by:
$$
d ^ {\pi} (s, \mathbf {a}) = (1 - \gamma) \sum_ {t = 0} ^ {\infty} \gamma^ {t} \Pr (s _ {t} = s, \mathbf {a} _ {t} = \mathbf {a}), \quad s _ {0} \sim p _ {0}, \mathbf {a} _ {t} \sim \pi (s _ {t}), s _ {t + 1} \sim P (s _ {t}, \mathbf {a} _ {t}), \forall t \geq 0.
$$
In offline MARL, online interaction with the environment is not allowed, and the policy is optimized only using the offline dataset  $D = \{(s, \mathbf{a}, r, s')_k\}_{k=1}^{|D|}$  collected by diverse data-collection agents. We denote the dataset distribution as  $d^{D}$  and abuse the notation  $d^{D}$  for  $s \sim d^{D}$ ,  $(s, a) \sim d^{D}$ , and  $(s, a, s') \sim d^{D}$ .
Nash Policy We focus on finding the factorizable policy  $\{\pi_i : \mathcal{S} \to \Delta(\mathcal{A}_i)\}_{i=1}^N \in \Pi_f$  during centralized training, which necessitate the notions of Nash and  $\epsilon$ -Nash policies.
Definition 2.1 (Nash and  $\epsilon$ -Nash Policy). A joint policy  $\pi^{*} = \langle \pi_{i}^{*}\rangle_{i = 1}^{N}$  is a Nash policy if it holds
$$
V _ {i} ^ {\pi_ {i} ^ {*}, \pi_ {- i} ^ {*}} (s) \geq V _ {i} ^ {\pi_ {i}, \pi_ {- i} ^ {*}} (s), \quad \forall i \in \mathcal {N}, \pi_ {i}, s \in \mathcal {S}. \tag {1}
$$
Similarly, a joint policy  $\pi^{*} = \langle \pi_{i}^{*}\rangle_{i = 1}^{N}$  is an  $\epsilon$ -Nash policy if there exists an  $\epsilon >0$  so that for each agent  $i\in \mathcal{N}$ , it holds that  $V_{i}^{\pi_{i}^{*},\pi_{-i}^{*}}(s)\geq V_{i}^{\pi_{i},\pi_{-i}^{*}}(s) - \epsilon$ , for all  $\pi_i,s$ . In other words,  $\pi^{*} = \langle \pi_{i}^{*},\pi_{-i}^{*}\rangle$  is a Nash policy, if each agent  $i\in \mathcal{N}$  has no incentive to deviate from  $\pi_i^*$  to an alternative policy,  $\pi_{i}$ , given that all other agents are playing  $\pi_{-i}^{*}$ .
3 AlberDICE
In this section, we introduce AlberDICE, an offline MARL algorithm that optimizes the stationary distribution of each agent's factorizable policy while effectively avoiding OOD joint actions. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time in an alternating manner. In contrast to existing methods that adopt decentralized training of each agent [25] where other agents' policies are assumed to follow the dataset distribution, AlberDICE adopts centralized training of each agent, which takes into account the non-stationarity incurred by the changes in other agents' policies.
Regularized Linear Program (LP) for MMDP The derivation of our algorithm starts by augmenting the standard linear program for MMDP with an additional KL-regularization term, where only a single agent  $i \in \mathcal{N}$  is being optimized while other agents' policies  $\pi_{-i}$  are fixed:
$$
\max  _ {d _ {i} \geq 0} \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} (s, a _ {i}) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s) r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \mathrm {D} _ {\mathrm {K L}} \left(d _ {i} (s, a _ {i}) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s) \| d ^ {D} (s, a _ {i}, \mathbf {a} _ {- i})\right) \tag {2}
$$
$$
\text {s . t .} \sum_ {a _ {i} ^ {\prime}, \mathbf {a} _ {- i} ^ {\prime}} d _ {i} \left(s ^ {\prime}, a _ {i} ^ {\prime}\right) \boldsymbol {\pi} _ {- i} \left(\mathbf {a} _ {- i} ^ {\prime} \mid s ^ {\prime}\right) = (1 - \gamma) p _ {0} \left(s ^ {\prime}\right) + \gamma \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} P \left(s ^ {\prime} \mid s, a _ {i}, \mathbf {a} _ {- i}\right) d _ {i} \left(s, a _ {i}\right) \boldsymbol {\pi} _ {- i} \left(\mathbf {a} _ {- i} \mid s\right) \forall s ^ {\prime}, \tag {3}
$$
where  $\mathrm{D}_{\mathrm{KL}}(p(x)\| q(x))\coloneqq \sum_{x}p(x)\log \frac{p(x)}{q(x)}$  is the KL-divergence between probability distributions  $p$  and  $q$ , and  $\alpha >0$  is a hyperparameter that controls the degree of conservatism, i.e., the amount of
penalty for deviating from the data distribution, which is a commonly adopted principle in offline RL [8, 14, 22, 42]. Satisfying the Bellman-flow constraints (3) guarantees that  $d(s,a_{i},\mathbf{a}_{-i}) \coloneqq d_{i}(s,a_{i})\pi_{-i}(\mathbf{a}_{-i}|s)$  is a valid stationary distribution in the MMDP.
As we show in our theoretical treatment of the regularized LP in Section 4, the selected regularization term defined in terms of joint action space critically ensures that every agent  $i \in \mathcal{N}$  optimizes the same objective function in (2). This ensures that when agents optimize alternately, the objective function always monotonically improves which, in turn, guarantees convergence (see Theorem 4.2). This is in contrast to existing methods such as [25], where each agent optimizes the different objective functions. Importantly, this is achieved while ensuring conservatism. As can be seen from (2), the KL-regularization term is defined in terms of the joint stationary distribution of all agents which ensures that the optimization of the regularized LP effectively avoids OOD joint action selection.
The optimal solution of the regularized LP (2-3),  $d_i^*$ , corresponds to the stationary distribution for a best response policy  $\pi_i^*$  against the fixed  $\pi_{-i}$ , and  $\pi_i^*$  can be obtained by  $\pi_i^* = \frac{d_i^*(s, a_i)}{\sum_{a_i'} d_i^*(s, a_i)}$ .
The (regularized) LP (2-3) can also be understood as solving a (regularized) reduced MDP  $\bar{M}_i = \langle S, \mathcal{A}_i, \bar{P}_i, \bar{r}_i, \gamma, p_0 \rangle$  for a single agent  $i \in \mathcal{N}$ , where  $\bar{P}_i$  and  $\bar{r}_i$  are defined as follows:
$$
\bar {P} _ {i} \left(s ^ {\prime} \mid s, a _ {i}\right) := \sum_ {\mathbf {a} _ {- i}} \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} \mid s) P \left(s ^ {\prime} \mid s, a _ {i}, \mathbf {a} _ {- i}\right), \quad \bar {r} _ {i} \left(s, a _ {i}\right) := \sum_ {\mathbf {a} _ {- i}} \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} \mid s) r \left(s, a _ {i}, \mathbf {a} _ {- i}\right).
$$
Then,  $d_{i}^{*}$  is an optimal stationary distribution on the reduced MDP,  $\bar{M}_{i}$ , but the reduced MDP is non-stationary due to other agents' policy,  $\pi_{-i}$ , updates. Therefore, it is important to account for changes in  $\pi_{-i}$  during training in order to avoid selection of OOD joint actions.
Lagrangian Formulation The constrained optimization (2-3) is not directly solvable since we do not have a white-box model for the MMDP. In order to make (2-3) amenable to offline learning in a model-free manner, we consider a Lagrangian of the constrained optimization problem:
$$
\begin{array}{l} \min_{\nu_{i}}\max_{d_{i}\geq 0}\mathbb{E}_{\substack{(s,a_{i})\sim d_{i}\\ \mathbf{a}_{-i}\sim \pi_{-i}(s)}}[r(s,a_{i},\mathbf{a}_{-i})] - \alpha \sum_{s,a_{i},\mathbf{a}_{-i}}d_{i}(s,a_{i})\pi_{-i}(\mathbf{a}_{-i}|s)\log \frac{d_{i}(s,a_{i})\pi_{-i}(\mathbf{a}_{-i}|s)}{d^{D}(s,a_{i})\pi_{-i}^{D}(\mathbf{a}_{-i}|s,a_{i})} \\ + \sum_ {s ^ {\prime}} \nu_ {i} (s ^ {\prime}) \left[ (1 - \gamma) p _ {0} (s ^ {\prime}) - \sum_ {a _ {i} ^ {\prime}} d _ {i} (s ^ {\prime}, a _ {i} ^ {\prime}) + \gamma \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} P (s ^ {\prime} | s, a _ {i}, \mathbf {a} _ {- i}) d _ {i} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \right] \tag {4} \\ \end{array}
$$
where  $\nu_{i}(s)\in \mathbb{R}$  is the Lagrange multiplier for the Bellman flow constraints. Still, (4) is not directly solvable due to its requirement of  $P(s^{\prime}|s,a_{i},\mathbf{a}_{-i})$  for  $(s,a_i)\sim d_i$  that are not accessible in the offline setting. To make progress, we re-arrange the terms in (4) as follows
$$
\begin{array}{l} \min  _ {\nu_ {i}} \max  _ {d _ {i} \geq 0} (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \mathbb {E} _ {(s, a _ {i}) \sim d _ {i}} \left[ - \alpha \log \frac {d _ {i} (s , a _ {i})}{d ^ {D} (s , a _ {i})} \right. (5) \\ + \underbrace {\mathbb {E} _ {\substack {\mathbf {a} _ {- i} \sim \boldsymbol {\pi} _ {- i} (s) \\ s ^ {\prime} \sim P (s , a _ {i} , \mathbf {a} _ {- i})}} \left[ r (s , a _ {i} , \mathbf {a} _ {- i}) - \alpha \log \frac {\boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s)}{\boldsymbol {\pi} _ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} + \gamma \nu_ {i} (s ^ {\prime}) - \nu_ {i} (s) \right]} _ {=: e _ {\nu_ {i}} (s, a _ {i})} \Bigg ] \\ = \min  _ {\nu_ {i}} \max  _ {d _ {i} \geq 0} (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \mathbb {E} _ {(s, a _ {i}) \sim d ^ {D}} \left[ \frac {d _ {i} (s , a _ {i})}{d ^ {D} (s , a _ {i})} \left(e _ {\nu_ {i}} (s, a _ {i}) - \alpha \log \underbrace {\frac {d _ {i} (s , a _ {i})}{d ^ {D} (s , a _ {i})}} _ {=: w _ {i} (s, a _ {i})}\right) \right] (6) \\ = \min  _ {\nu_ {i}} \max  _ {w _ {i} \geq 0} (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \mathbb {E} _ {(s, a _ {i}) \sim d ^ {D}} \left[ w _ {i} (s, a _ {i}) \left(e _ {\nu_ {i}} (s, a _ {i}) - \alpha \log w _ {i} (s, a _ {i})\right) \right] (7) \\ \end{array}
$$
where  $e_{\nu_i}(s, a_i)$  is the advantage by  $\nu_i$ , and  $w_i(s, a_i)$  are the stationary distribution correction ratios between  $d_i$  and  $d^D$ . Finally, to enable every term in (7) to be estimated from samples in the offline dataset  $D$ , we adopt importance sampling, which accounts for the distribution shift in other agents' policies,  $\pi_{-i}$ :
$$
\begin{array}{l} \min_{\nu_{i}}\max_{w_{i}\geq 0}(1 - \gamma)\mathbb{E}_{p_{0}}[\nu_{i}(s_{0})] + \\ + \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i} s ^ {\prime}) \sim d ^ {D}} \left[ w _ {i} (s, a _ {i}) \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} \left(\hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - \alpha \log w _ {i} (s, a _ {i})\right) \right] \tag {8} \\ \end{array}
$$
5The reduced MDP is also used by [44], where it is termed averaged MDP.
The use of  $\min_{\nu_i}\max_{d_i\geq 0}$  rather than  $\max_{d_i\geq 0}\min_{\nu_i}$  is justified due to the convexity of the optimization problem in (2) which allows us to invoke strong duality and Slater's condition.
where  $\hat{e}_{\nu_i}(s,a_i,\mathbf{a}_{-i},s') := r(s,a_i,\mathbf{a}_{-i}) - \alpha \log \frac{\pi_{-i}(\mathbf{a}_{-i}|s)}{\pi_{-i}^D(\mathbf{a}_{-i}|s,a_i)} + \gamma \nu_i(s') - \nu_i(s)$ . Every term in (8) can be now evaluated using only the samples in the offline dataset. Consequently, AlberDICE aims to solve the unconstrained minimax optimization (8) for each agent  $i \in \mathcal{N}$ . Once we compute the optimal solution  $(\nu_i^*,w_i^*)$  of (8), we obtain the information about the optimal policy  $\pi_i^*$  (i.e. the best response policy against the fixed  $\pi_{-i}$ ) in the form of distribution correction ratios  $w_i^* = \frac{d^{\pi_i^*}(s,a_i)}{d^D(s,a_i)}$ .
Pretraining autoregressive data policy To optimize (8), we should be able to evaluate  $\pi_{-i}^{D}(\mathbf{a}_{-i}|s,a_{i})$  for each  $(s,a_i,\mathbf{a}_{-i})\in D$ . To this end, we pretrain the data policy via behavior cloning, where we adopt an MLP-based autoregressive policy architecture, similar to the one in [43]. The input dimension of  $\pi_{-i}^{D}$  only grows linearly with the number of agents. Then, for each  $i\in \mathcal{N}$  we optimize the following:
$$
\max  _ {\boldsymbol {\pi} _ {- i} ^ {D}} \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}) \sim d ^ {D}} \left[ \sum_ {j = 1, j \neq i} ^ {N} \log \boldsymbol {\pi} _ {- i} ^ {D} \left(a _ {j} \mid s, a _ {i}, a _ {<   j}\right) \right] \tag {9}
$$
While, in principle, the joint action space grows exponentially with the number of agents, learning a joint data distribution in an autoregressive manner is known to work quite well in practice [7, 29].
Practical Algorithm: Minimax to Min Still, solving the nested minimax optimization (7) can be numerically unstable in practice. In this section, we derive a practical algorithm that solves a single minimization only using offline samples. For brevity, we denote each sample  $(s,a_i,\mathbf{a}_{-i},s')$  in the dataset as  $x$ . Also, let  $\hat{\mathbf{E}}_{x\in D}[f(x)] := \frac{1}{|D|}\sum_{x\in D}f(x)$  be a Monte-Carlo estimate of  $\mathbb{E}_{x\sim p}[f(x)]$ , where  $D = \{x_k\}_{k = 1}^{|D|}\sim p$ . First, we have an unbiased estimator of (7):
$$
\min  _ {\nu_ {i}} \max  _ {w _ {i} \geq 0} (1 - \gamma) \hat {\mathbf {E}} _ {s _ {0} \in D _ {0}} [ \nu_ {i} (s _ {0}) ] + \hat {\mathbf {E}} _ {x \in D} \left[ w _ {i} (s, a _ {i}) \rho_ {i} (x) \left(\hat {e} _ {\nu_ {i}} (x) - \alpha \log w _ {i} (s, a _ {i})\right) \right] \tag {10}
$$
where  $\rho_{i}(x)$  is defined as:
$$
\rho_ {i} (x) := \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} = \frac {\prod_ {j \neq i} \pi_ {j} \left(a _ {j} \mid s\right)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})}. \tag {11}
$$
Optimizing (10) can suffer from large variance due to the large magnitude of  $\rho(x)$ , which contains products of  $N - 1$  policies. To remedy the large variance issue, we adopt Importance Resampling (IR) [32] to (10). Specifically, we sample a mini-batch of size  $K$  from  $D$  with probability proportional to  $\rho(x)$ , which constitutes a resampled dataset  $D_{\rho_i} = \{(s, a_i, \mathbf{a}_{-i}, s')_k\}_{k=1}^K$ . Then, we solve the following optimization, which now does not involve the importance ratio:
$$
\min  _ {\nu_ {i}} \max  _ {w _ {i} \geq 0} (1 - \gamma) \hat {\mathbf {E}} _ {s _ {0} \in D _ {0}} [ \nu_ {i} (s _ {0}) ] + \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \left[ w _ {i} (s, a _ {i}) \left(\hat {e} _ {\nu_ {i}} (x) - \alpha \log w _ {i} (s, a _ {i})\right) \right] \tag {12}
$$
where  $\bar{\rho}_i\coloneqq \hat{\mathbf{E}}_{x\in D}[\rho_i(x)]$ . It can be proven that (12) is still an unbiased estimator of (7) thanks to the bias correction term of  $\bar{\rho}$  [32]. The resampling procedure can be understood as follows: for each data sample  $x = (s,a_i,\mathbf{a}_{-i},s')$ , if other agents' policy  $\pi_{-i}$  selects the action  $\mathbf{a}_{-i}\in D$  with low probability, i.e.,  $\pi_{-i}(\mathbf{a}_{-i}|s)\approx 0$ , the sample  $x$  will be removed during the resampling procedure, which makes the samples in the resampled dataset  $D_{\rho_i}$  consistent with the reduced MDP  $\bar{M}_i$  's dynamics. Finally, to avoid the numerical instability associated with solving a min-max optimization problem, we exploit the properties of the inner-maximization problem in (12), specifically, its concavity in  $w_{i}$ , and derive its closed-form solution.