SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark D.1. In the LP formulation of AlberDICE, cf. equation (2) and (3), we used the notation  $d_{i}(s,a_{i})$  rather than  $d_{i}^{\pi}(s,a_{i})$ , since, in this case, the variables are  $d_{i}(s,a_{i})$  are decision variables that are not a-priori related to any particular policy  $\pi_{i}$ . However, once the  $d_{i}(s,a_{i})$ 's are fixed and translated to a policy, e.g., through the relation  $\pi_{i}(a_{i}|s) = \frac{d_{i}(s,a_{i})}{\sum_{a_{j}}d_{i}(s,a_{j})}$  that holds in tabular domains, then, we can apply Lemma 4.1. For the purposes of the alternating optimization procedure of the AlberDICE algorithm, Lemma 4.1 ensures that after each update by any agent  $i \in N$ , the value of the objective function is the same for each agent. To evaluate the modified utilities,  $\tilde{r}(s,a_i,\mathbf{a}_{-i})$ , during training, agents need to know both the action,  $a_{i}$ , and the policy,  $\pi_{i}(a,s)$ , from which its action drawn for each agent  $i \in N$ . Thus, this regularization terms exploits to the fullest knowledge available to agents in the centralized training setting.
To proceed, we can define a modified game,  $\tilde{G} = \langle N,S,\mathcal{A},\tilde{r},P,\gamma \rangle$  which is the same as the original game in every respect, i.e., it has the same state space, agents, actions, transitions and discount factor, except for the rewards,  $r$ , which are replaced by the modified rewards  $\tilde{r}$  for any given value of the regularization parameter,  $\alpha >0$  (for  $\alpha = 0$ , we simply have the rewards,  $r$ , of the original game,  $G$ ). Despite this modification, Lemma 4.1 implies that  $\tilde{G}$  still has a common reward structure. This can be used to prove that the AlberDICE algorithm has monotonic updates which eventually converge to a Nash equilibrium of the modified game,  $\tilde{G}$ . For this part, we focus on tabular domains, in which the policies,  $\pi_i(a_i\mid s)$ , can be directly extracted from  $d_{i}(s,a_{i})$  as  $\pi_i(a_i\mid s) = \frac{d_i(s,a_i)}{\sum_{a_j}d_i(s,a_j)}$ .
Proof of Theorem 4.2. Let  $\pi^0 = \langle \pi_i^0\rangle_{i\in N}$  denote the initial joint policy and let  $\pi^t = \langle \pi_i^t\rangle_{i\in N}$  denote the joint policy after iteration  $t\in \mathbb{N}$  in an execution of the AlberDICE algorithm. For  $i = 1,\ldots ,N$  we will also write  $\pi_{1:i}^{t}$  to denote the joint policy at time  $t$  after players 1 to  $i$  have updated their policies, i.e.,  $\pi_{1:i}^{t} = (\pi_{1}^{t},\dots ,\pi_{i}^{t},\pi_{i + 1}^{t - 1},\dots ,\pi_{N}^{t - 1})$ . Let  $d_{i}(s,a_{i})\coloneqq d_{i}^{\pi_{1:i - 1}^{t - 1}}(s,a_{i})$  denote the stationary distribution for agent  $i$  before the current optimization by player  $i$ , and let  $d_{i}^{*}(s,a_{i}):= d_{i}^{\pi_{1:i}^{t - 1}}(s,a_{i}),\pi_{i}^{*}(a_{i}\mid s)$  denote the stationary policy derived as the optimal solution of the LP and the corresponding extracted policy for agent  $i$ , respectively, after the optimization by agent  $i$  at time  $t$ . Then,
$$
\begin{array}{l} \tilde {V} _ {i} ^ {\pi_ {1: i}} (s) = \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {*} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} \mid s) \tilde {r} (s, a _ {i}, \mathbf {a} _ {- i}) \\ \geq \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} \mid s) \tilde {r} (s, a _ {i}, \mathbf {a} _ {- i}) = \tilde {V} _ {i} ^ {\pi_ {1: i - 1}} (s), \\ \end{array}
$$
for each state  $s \in S$ , where we used Lemma 4.1 for the equality of the modified rewards among all agents in  $N$ . The inequality is strict unless agent  $i$  is already using an optimal policy,  $\pi_i$ , against  $\pi_{-i}$ . Letting  $\tilde{V}$  to denote the common value function, i.e.,  $\tilde{V}_i \equiv \tilde{V}$  for all agents  $i \in N$ , then, the previous inequality implies that after the update of agent  $i$ , all agents have a higher value with the current policy  $\pi_{1:i}^t$ . Thus, the sequence,  $\pi^t$ , of joint policies generated by the AlberDICE algorithm results in monotonic updates (increases) in the joint modified value function  $\tilde{V}(s)$ ,  $s \in S$ . Since,  $V$  is bounded (rewards are bounded and discounted by assumption), this implies that also  $\tilde{V}$  is bounded and hence, at some point, the updates of the algorithm will reach a local maximum of  $V$ . Let  $\pi^* = (\pi_i^*, \pi_{-i}^*)$  denote the extracted policy at that point. Then, for all agents  $i \in N$  and any  $\pi_i$ , it holds that  $\tilde{V}_i^{\pi^*}(s) = \tilde{V}^{\pi^*} \geq \tilde{V}^{(\pi_i, \pi_{-i}^*)} = \tilde{V}_i^{(\pi_i, \pi_{-i}^*)}$  for all  $s \in S$ . Thus,  $\pi^*$  is a Nash policy as claimed.
An important property of the AlberDICE algorithm is that it maintains a sequence of factorizable joint policies,  $\pi^t = \langle \pi_i^k\rangle_{i\in N}$ , for any  $t > 0$ . Thus, after termination of the algorithm, the agents are guaranteed not only to reach an optimal state of the value function, but also an extracted joint policy that will be factorizable. This eliminates the problem of learning correlated policies during centralized training, which even if optimal, may not be useful during decentralized execution. This property of the AlberDICE is formally stated of Corollary 4.3. Its proof is immediate by the construction of the algorithm.
E Detailed Description of Practical Algorithm
E.1 Numerically Stable Optimization
The practical issue in optimizing (14) is that it is unstable due to its inclusion of  $\exp (\cdot)$ , often causing exploding gradient problems:
$$
L \left(\nu_ {i}\right) := \min  _ {\nu_ {i}} \bar {\rho} _ {i} \alpha \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) - 1\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} \left[ \nu_ {i} \left(s _ {0}\right) \right]. \tag {14}
$$
where  $\hat{e}_{\nu_i}(s,a_i,\mathbf{a}_{-i},s') := r(s,a_i,\mathbf{a}_{-i}) - \alpha \log \frac{\pi_{-i}(\mathbf{a}_{-i}|s)}{\pi_{-i}^D(\mathbf{a}_{-i}|s,a_i)} + \gamma \nu_i(s') - \nu_i(s)$ . To address this issue, we use the following alternative, which can be optimized stably.
$$
\tilde {\mathcal {L}} (\tilde {\nu} _ {i}) := \min  _ {\tilde {\nu} _ {i}} \alpha \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\tilde {\nu} _ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \tilde {\nu} _ {i} (s _ {0}) ] \tag {31}
$$
Note that the gradient  $\nabla_{x}\log \hat{\mathbf{E}}_{x}[\exp (h(x))]$  is given by  $\hat{\mathbf{E}}_x\left[\frac{\exp(h(x))}{\hat{\mathbf{E}}_{x'}[\exp(h(x'))]}\nabla_xh(x)\right]$ , which normalizes the value of  $\exp (h(x))$  and thus it is numerically stable by preventing the exploding gradient issue. At first glance, it seems that optimizing (31) can result in a completely different solution to the solution of (14). However, as we will show in the followings, their optimal objective function values are the same and their optimal solutions only differ in a constant shift.
Proposition E.1. Let  $V^{*} = \arg \min_{\nu_{i}} L(\nu_{i})$  and  $\widetilde{V}^{*} = \arg \min_{\tilde{\nu}_{i}} \tilde{\mathcal{L}}(\tilde{\nu}_{i})$  be the sets of optimal solutions of (14) and (31). Then,  $L(\nu_{i}^{*}) = \tilde{\mathcal{L}}(\tilde{\nu}_{i}^{*})$  holds for any  $\nu_{i}^{*} \in V^{*}$  and  $\tilde{\nu}_{i}^{*} \in \widetilde{V}^{*}$ . Also, for any  $\nu_{i}^{*} \in V$  and any  $C \in \mathbb{R}$ ,  $v_{i}^{*} + C \in \widetilde{V}^{*}$ .
We follow the proof steps in DemoDICE [9]. First, note that for any constant  $C$ , the advantage for  $\nu_{i} + C$  is:
$$
\begin{array}{l} \hat {e} _ {\nu_ {i} + C} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) = r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \log \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} + \gamma (\nu_ {i} (s ^ {\prime}) + C) - (\nu_ {i} (s) + C) \\ = \hat {e} _ {\nu_ {i}} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) - (1 - \gamma) C \tag {32} \\ \end{array}
$$
Lemma E.2. For an arbitrary function  $\nu_{i}$  and any constant  $C$ , the following equality holds,
$$
\tilde {\mathcal {L}} (\nu_ {i}) = \tilde {\mathcal {L}} (\nu_ {i} + C).
$$
Proof. From the definition of  $\tilde{\mathcal{L}} (\nu_i)$
$$
\begin{array}{l} \tilde {\mathcal {L}} (\nu_ {i} + C) \\ = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) + C ] + \alpha \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \Big [ \exp \Big (\frac {1}{\alpha} \hat {e} _ {\nu_ {i} + C} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \Big) \Big ] \\ = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) + C ] + \alpha \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - \frac {1}{\alpha} (1 - \gamma) C\right) \right] \tag {by(32)} \\ = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) + C ] + \alpha \log \left\{\exp \left(- \frac {1}{\alpha} (1 - \gamma) C\right) \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \right\} \\ = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) + C ] - (1 - \gamma) C + \alpha \log \bar {\rho_ {i}} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \Big [ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \Big ] \\ = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \alpha \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ = \tilde {\mathcal {L}} (\nu_ {i}) \\ \end{array}
$$
Lemma E.3. For any function  $\nu_{i}$ , the following inequality always holds:
$$
L (\nu_ {i}) \geq \tilde {\mathcal {L}} (\nu_ {i}).
$$
Equality holds if and only if
$$
\bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] = 1
$$
Proof. For any  $y \geq 0$
$$
y - 1 \geq \log y
$$
and equality holds if and only if  $y = 1$ . Thus,
$$
\bar {\rho} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \Big [ \exp \Big (\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \Big) \Big ] - 1 \geq \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \Big [ \exp \Big (\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \Big) \Big ]
$$
and equality holds if and only if
$$
\bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \Big [ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \Big ] = 1.
$$
Finally, we obtain the following results:
$$
\begin{array}{l} \tilde {L} (\nu_ {i}) = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \bar {\rho} _ {i} \alpha \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - 1\right) \right] \\ \geq (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \alpha \log \left\{\bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - 1\right) \right] \right\} + 1 \\ \geq (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \alpha \log \left\{\bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - 1\right) \right] \exp (1) \right\} \\ = (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] + \alpha \log \left\{\bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \Big [ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \Big ] \right\} \\ =: \tilde {\mathcal {L}} (\nu_ {i}) \\ \end{array}
$$
Lemma E.4. For any optimal solution  $\tilde{\nu}_i^* = \arg \min_{\nu_i} \tilde{\mathcal{L}}(\nu_i)$ , there is a constant  $C$  such that  $\tilde{\nu}_i^* + C$  is an optimal solution of  $\min_{\nu_i} L(\nu_i)$ .
Proof. Let  $\tilde{\nu}_i^*$  be an optimal solution of  $\arg \min_{\nu_i}\tilde{\mathcal{L}} (\nu_i)$  and
$$
C ^ {*} := \frac {\alpha}{1 - \gamma} \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right]
$$
Then,  $\hat{\nu}_i\coloneqq \tilde{\nu}_i^* +C^*$  satisfies
$$
\begin{array}{l} \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\hat {\nu} _ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ = \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\tilde {\nu} _ {i} ^ {*} + C ^ {*}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ = \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\tilde {\nu} _ {i} ^ {*}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - \frac {1 - \gamma}{\alpha} C ^ {*}\right) \right] \\ = \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\tilde {\nu} _ {i} ^ {*}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \exp \left(- \frac {1 - \gamma}{\alpha} C ^ {*}\right) \right] \\ = 1. \\ \end{array}
$$
Furthermore,  $\hat{\nu}_i$  is also an optimal solution of  $\min_{\nu_i}\tilde{\mathcal{L}} (\nu_i)$  by Lemma E.2 . Then, by the equality condition in Lemma E.3 ,
$$
\tilde {L} (\hat {\nu} _ {i}) = \tilde {\mathcal {L}} (\hat {\nu} _ {i}) = \min  _ {\nu_ {i}} \tilde {\mathcal {L}} (\nu_ {i}) \leq \min  _ {\nu_ {i}} \tilde {L} (\nu_ {i}).
$$
Thus,  $\hat{\nu}_i$  is an optimal solution of  $\min_{\nu_i} L(\nu_i)$ .
Lemma E.5. An optimal solution  $\nu_{i}^{*} = \arg \min_{\nu_{i}}L(\nu_{i})$  is also an optimal solution of  $\min_{\nu_i}\tilde{\mathcal{L}} (\nu_i)$
Proof. From Lemma E.3,
$$
\min  _ {\nu_ {i}} L \left(\nu_ {i}\right) = L \left(\nu_ {i} ^ {*}\right) \geq \tilde {\mathcal {L}} \left(\nu_ {i} ^ {*}\right).
$$
From Lemma E.4,  $\min_{\nu_i} L(\nu_i)$  and  $\min_{\nu_i} \tilde{\mathcal{L}}(\nu_i)$  have the same minimum value, and thus,
$$
\tilde {\mathcal {L}} \left(\nu_ {i} ^ {*}\right) \leq L \left(\nu_ {i} ^ {*}\right) = \min  _ {\nu_ {i}} L \left(\nu_ {i}\right) = \min  _ {\nu_ {i}} \tilde {\mathcal {L}} \left(\nu_ {i}\right)
$$
The aforementioned Proposition E.1 can now be proved.
Proof. This holds from combining Lemma E.4 and Lemma E.5
E.2 Policy Extraction
Finally, our practical AlberDICE optimizes (31) that yields  $\tilde{\nu}_i^*$ . However,  $\tilde{\nu}_i^*$  itself is not a directly executable policy, so we should extract a policy from it. To this end, first note that the optimal policy  $\pi_i^*$  is encoded in  $w_i^*$  as a form of stationary distribution correction ratios:
$$
w _ {i} ^ {*} (s, a) = \frac {d ^ {\pi_ {i} ^ {*}} (s , a _ {i})}{d ^ {D} (s , a _ {i})} \quad (\text {o p t i m a l s o l u t i o n o f E q . (8)}) \tag {33}
$$
Then,  $w_{i}^{*}(s,a)$  can be represented in terms of  $\tilde{\nu}_i^*$  as follows:
$$
w _ {i} ^ {*} (s, a) = \exp \left(\frac {1}{\alpha} e _ {\nu_ {i} ^ {*}} (s, a _ {i}) - 1\right) \quad (\text {E q .} (2 2)) \tag {34}
$$
$$
\propto \exp \left(\frac {1}{\alpha} e _ {\tilde {\nu} _ {i} ^ {*}} (s, a _ {i})\right) \quad (\text {b y P r o p i s t i o n E . 1}) \tag {35}
$$
where  $e_{\tilde{\nu}_i^*}(s,a_i) = \mathbb{E}_{\substack{\mathbf{a}_{-i}\sim \boldsymbol {\pi}_{-i}(s)\\ s'\sim P(s,a_i,\mathbf{a}_{-i})}}\left[\hat{e}_{\tilde{\nu}_i^*}(s,a_i,\mathbf{a}_{-i},s')\right]$  and  $\hat{e}_{\tilde{\nu}_i^*}(s,a_i,\mathbf{a}_{-i},s') = r(s,a_i,\mathbf{a}_{-i}) -$
$\alpha \log \frac{\pi_{-i}(\mathbf{a}_{-i}|s)}{\pi_{-i}^D(\mathbf{a}_{-i}|s,a_i)} + \gamma \tilde{\nu}_i^* (s') - \tilde{\nu}_i^* (s)$ . Finally, we extract a policy from  $w_{i}^{*}$  via I-projection policy extraction method introduced in equation (15).
$$
\min  _ {\pi_ {i}} \mathrm {D} _ {\mathrm {K L}} \left(d ^ {D} (s) \pi_ {i} \left(a _ {i} | s\right) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s) \mid \mid d ^ {D} (s) \pi_ {i} ^ {*} \left(a _ {i} | s\right) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s)\right) \tag {36}
$$
$$
= \hat {\mathbf {E}} _ {s \in D, a _ {i} \sim \pi_ {i}, \mathbf {a} _ {- i} \sim \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s)} \left[ \log \frac {d ^ {D} (s) \pi_ {i} (a _ {i} | s) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s)}{d ^ {D} (s) \pi_ {i} ^ {*} (a _ {i} | s) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} | s)} \right] \tag {37}
$$
$$
= \hat {\mathbf {E}} _ {s \sim D, a _ {i} \sim \pi_ {i}} \left[ \log \frac {d ^ {D} (s) \pi_ {i} ^ {D} \left(a _ {i} \mid s\right)}{d ^ {\pi_ {i} ^ {*}} (s) \pi_ {i} ^ {*} \left(a _ {i} \mid s\right)} + \log \frac {\pi_ {i} \left(a _ {i} \mid s\right)}{\pi_ {i} ^ {D} \left(a _ {i} \mid s\right)} + \underbrace {\log \frac {d ^ {\pi_ {i} ^ {*}} (s)}{d ^ {D} (s)}} _ {\text {c o n s t a n t f o r} \pi_ {i}} \right] \tag {38}
$$
$$
= \hat {\mathbf {E}} _ {s \in D, a _ {i} \sim \pi_ {i}} \left[ \log \frac {d ^ {D} (s , a _ {i})}{d ^ {\pi_ {i} ^ {*}} (s , a)} + \mathrm {D} _ {\mathrm {K L}} \left(\pi_ {i} \left(a _ {i} | s\right) \mid \mid \pi_ {i} ^ {D} \left(a _ {i} | s\right)\right) \right] + C _ {1} \tag {39}
$$
$$
= \hat {\mathbf {E}} _ {s \in D, a _ {i} \sim \pi_ {i}} \left[ - \log w _ {i} ^ {*} (s, a _ {i}) + \mathrm {D} _ {\mathrm {K L}} \left(\pi_ {i} \left(a _ {i} | s\right) \mid \mid \pi_ {i} ^ {D} \left(a _ {i} | s\right)\right) \right] + C _ {1} \tag {40}
$$
$$
= \hat {\mathbf {E}} _ {s \in D, a _ {i} \sim \pi_ {i}} \left[ - \frac {1}{\alpha} e _ {\tilde {\nu} _ {i} ^ {*}} (s, a _ {i}) + \mathrm {D} _ {\mathrm {K L}} \left(\pi_ {i} \left(a _ {i} | s\right) \mid \mid \pi_ {i} ^ {D} \left(a _ {i} | s\right)\right) \right] + C _ {1} + C _ {2} \quad (\text {b y} (3 5)) \tag {41}
$$
where  $C_1$  and  $C_2$  denote some constants.  $\pi_i^D (a_i|s)$  is a data policy for  $i$ -th agent, which is pretrained by maximizing the log-likelihood:
$$
\max  _ {\pi_ {i} ^ {D}} \hat {\mathbf {E}} _ {(s, a _ {i}) \in D} [ \log \pi_ {i} ^ {D} (a _ {i} | s) ] \tag {42}
$$
Eq. (41) can be understood as KL-regularized policy optimization, where we aim to maximize  $e_{\tilde{\nu}_i^*}(s,a_i)$  (analogous to critic value) while not deviating too much from the data policy, whose trade-off is controlled by the hyperparameter  $\alpha$ . Finally, to enable  $e_{\tilde{\nu}_i^*}$  to be evaluated at every action  $a_i$ , we train an additional parametric function  $e_i$  (implemented as an MLP that takes  $(s,a_i)$  as an input and outputs a scalar value) by minimizing the mean squared error with a conservative regularization term  $\mathcal{R}(e_i)$  introduced in CQL [11] to penalize OOD action values:
$$
\min  _ {e _ {i}} \hat {\mathbf {E}} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \in D _ {\rho_ {i}}} \left[ \left(e _ {i} (s, a _ {i}) - \hat {e} _ {\tilde {\nu} _ {i} ^ {*}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) ^ {2} \right] + \mathcal {R} _ {\mathrm {C Q L}} \left(e _ {i}\right) \tag {43}
$$
where  $\mathcal{R}_{\mathrm{CQL}}(e_i) := \alpha_{\mathrm{CQL}} \hat{\mathbf{E}}_{(s,a_i) \sim D_{\rho_i}}[\log \sum_{a_i} \exp(e_i(s,a_i)) - \mathbb{E}_{a_i \sim \pi_i^D}[e_i(s,a_i)]]$ . We used  $\alpha_{\mathrm{CQL}} = 0.1$  for all experiments.
E.3 Pseudocode of AlberDICE
To sum up, AlberDICE computes the best response of agent  $i$  by optimizing  $\nu_{i}$ , which corresponds to obtaining a stationary distribution correction ratios of the optimal policy. Then, we extract a policy by training  $e$ -network and performing I-projection as described in Section E.2.
We assume  $\pi_i^D$ ,  $\pi_{-i}^D$ ,  $\nu_{i}$ ,  $e_i$ , and  $\pi_{i}$  are parameterized by  $\beta_{i}$ ,  $\beta_{-i}$ ,  $\theta_{i}$ ,  $\psi_{i}$ , and  $\phi_{i}$ , respectively<sup>10</sup>. Then, we optimize the parameters via stochastic gradient descent (SGD). The entire loss functions to optimize the parameters are summarized in the following:
$$
J \left(\beta_ {i}\right) := - \hat {\mathbf {E}} _ {(s, a _ {i}) \in D} \left[ \log \pi_ {\beta_ {i}} ^ {D} \left(a _ {i} \mid s\right) \right] \tag {44}
$$
$$
J \left(\beta_ {- i}\right) := - \hat {\mathbf {E}} _ {\left(s, a _ {i}, \mathbf {a} _ {- i}\right) \in D} \left[ \sum_ {j = 1, j \neq i} ^ {N} \log \pi_ {\beta_ {- i}} ^ {D} \left(a _ {j} \mid s, a _ {i}, a _ {<   j}\right) \right] \tag {45}
$$
$$
J \left(\theta_ {i}\right) := \alpha \log \bar {\rho} _ {i} \hat {\mathbf {E}} _ {x \sim D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {\theta_ {i}}} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right)\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} \left[ \nu_ {\theta_ {i}} \left(s _ {0}\right) \right] \tag {46}
$$
$$
\begin{array}{l} J \left(\psi_ {i}\right) := \hat {\mathbf {E}} _ {\left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) \in D _ {\rho_ {i}}} \left[ \left(e _ {\psi_ {i}} (s, a _ {i}) - \hat {e} _ {\tilde {\nu} _ {\theta_ {i}}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) ^ {2} \right] \tag {47} \\ + \alpha_ {\mathrm {C Q L}} \hat {\mathbf {E}} _ {s \in D _ {\rho_ {i}}} \left[ \left(\log \sum_ {a _ {i}} \exp \left(e _ {\psi_ {i}} (s, a _ {i})\right) - \mathbb {E} _ {a _ {i} \sim \pi_ {\beta_ {i}} ^ {D}} [ e _ {\psi_ {i}} (s, a _ {i}) ]\right) \right] \\ \end{array}
$$
$$
J \left(\phi_ {i}\right) := \hat {\mathbf {E}} _ {s \in D} \left[ \sum_ {a _ {i}} \pi_ {\phi_ {i}} \left(a _ {i} \mid s\right) \left(- e _ {\psi_ {i}} \left(s, a _ {i}\right) + \alpha \log \frac {\pi_ {\phi_ {i}} \left(a _ {i} \mid s\right)}{\pi_ {\beta_ {i}} ^ {D} \left(a _ {i} \mid s\right)}\right) \right] \tag {48}
$$
The pseudocode of AlberDICE is presented in Algorithm 1.
Algorithm 1 AlberDICE
Input: A dataset  $D \coloneqq \{(s, \mathbf{a}, r, s')_k\}_{k=1}^{|D|}$ , a set of initial states  $D_0 \coloneqq \{s_{0,k}\}_{k=1}^{|D_0|}$ , data policy networks  $\{(\pi_{\beta_i}^D, \pi_{\beta_{-i}}^D)\}_{i=1}^N$  with parameters  $\{(\beta_i, \beta_{-i})\}_{i=1}^N$ ,  $\nu$ -networks  $\{\nu_{\theta_i}\}_{i=1}^N$  with parameters  $\{\theta_i\}_{i=1}^N$ ,  $e$ -networks  $\{e_{\psi_i}\}_{i=1}^N$  with parameters  $\{\psi_i\}_{i=1}^N$ , policy networks  $\{\pi_i^\phi\}_{i=1}^N$  with parameters  $\{\phi_i\}_{i=1}^N$ , and a learning rate  $\eta$
1: Pretrain (auto-regressive) data policies  $\left\{\left(\pi_{\beta_i}^D(a_i|s), \pi_{\beta_{-i}}^D(\mathbf{a}_{-i}|s, a_i)\right)\right\}_{i=1}^N$  by minimizing (44-45).
2: for each iteration until convergence do
3: for each agent  $i\in \mathcal{N}$  do
4: Sample mini-batches from  $s_0 \sim D_0$  and  $x \sim D$ .
5: Compute the importance ratio  $\rho_{i}(x) = \frac{\prod_{j\neq i}\pi_{\phi_{j}}(a_{j}|s)}{\pi_{\beta - i}^{D}(\mathbf{a}_{-i}|s,a_{i})}$  for each sample  $x$  (Eq. (11))
6: Perform resampling with probability proportional to  $\rho_{i}(x)$ , which constitutes the resampled dataset  $D_{\rho_i}$ .
7: Perform SGD updates using  $D_0$  and  $D_{\rho_i}$
$$
\theta_ {i} \leftarrow \theta_ {i} - \eta \nabla_ {\theta_ {i}} J (\theta_ {i}) \quad \text {(E q . (4 6))}
$$
$$
\psi_ {i} \leftarrow \psi_ {i} - \eta \nabla_ {\psi_ {i}} J (\psi_ {i}) \quad (E q. (4 7))
$$
$$
\phi_ {i} \leftarrow \phi_ {i} - \eta \nabla_ {\phi_ {i}} J (\psi_ {i}) \quad (E q. (4 8))
$$
8: end for
Output: Factorized policies  $\{\pi_{\phi_i}(a_i|s)\}_{i = 1}^N$
To increase scalability, we use shared parameters and an additional agent ID input to train  $\beta_{i}$  for  $\pi_i^D$  in all experiments.
F Dataset Details
F.1 Bridge