SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Dimitri Bertsekas. Multiagent value iteration algorithms in dynamic programming and reinforcement learning, 2020.
[2] Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2020.
[3] A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open Problems in Cooperative AI. arXiv e-prints, December 2020.
[4] A. Dafoe, Y. Bachrach, G. Hadfield, E. Horvitz, K. Larson, and T. Graepel. Cooperative AI: machines must learn to find common ground. Nature, 7857:33-36, 2021. doi: 10.1038/d41586-021-01170-0.
[5] Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 6863-6877. PMLR, 17-23 Jul 2022.
[6] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR, 09-15 Jun 2019.
[7] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, volume 34, pages 1273-1286. Curran Associates, Inc., 2021.
[8] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MReL: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[9] Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022.
[10] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations, 2022.
[11] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191. Curran Associates, Inc., 2020.
[12] K. Kurach, A. Raichuk, P. Stańczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, and S. Gelly. Google research football: A novel reinforcement learning environment, 2019.
[13] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning, pages 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
[14] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6120-6130. PMLR, 18-24 Jul 2021.
[33] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5887-5896. PMLR, 09-15 Jun 2019.
[34] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.
[35] Wei-Cheng Tseng, Tsun-Hsuan Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. In Advances in Neural Information Processing Systems, 2022.
[36] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multiagent q-learning. In International Conference on Learning Representations, 2021.
[37] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. In Advances in Neural Information Processing Systems, 2022.
[38] C. S. D. Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? ArXiv, abs/2011.09533, 2020.
[39] Erfu Yang and Dongbing Gu. Multiagent reinforcement learning for multi-robot systems: A survey. 06 2004.
[40] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 10299-10312. Curran Associates, Inc., 2021.
[41] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[42] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[43] Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, ziyu wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2021.
[44] Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity, 2021.
A Individual-Global-Max (IGM) and its Limitations
Centralized Training with Decentralized Execution (CTDE) refers to a paradigm in MARL where the training phase is permitted to utilize any global information such as the joint policy  $\pi = \langle \pi_1, \dots, \pi_N \rangle$  and global state  $s$ . Practical CTDE algorithms avoid the combinatorial nature of the joint action space, and reduce the non-stationarity, which arises from agents' simultaneous policy updates during training.
However, an important challenge for CTDE algorithms is that during training they still need to learn a joint policy that can be factorized into individual policies. During the execution phase, agents take individual actions,  $a_{i}$ , without conditioning on other agents' actions,  $\mathbf{a}_{-i}$ , or policies,  $\pi_{-i}$ . This independence assumption poses a challenge for many pre-existing algorithms, especially for offline MARL when the dataset is generated by a mixture of different data collection policies and/or the environment contains multiple global optima.
One popular way to learn factorizable policies from centralized training is to impose an Individual-Global-Max (IGM) assumption.
Definition A.1 (Individual-Global-Max (IGM) [30, 33]). Individual utility functions  $\{Q_i\}_{i=1}^N$  satisfies the IGM condition for a joint state-action value function  $Q: S \times \mathcal{A} \to \mathbb{R}$  if the following condition holds:
$$
\arg \max  _ {a} Q (s, a) = \left\{\arg \max  _ {i = 1} Q _ {i} (s, a _ {i}) \right\} _ {i = 1} ^ {N} \tag {17}
$$
Intuitively, the IGM condition implies that the optimal Q-function  $Q^{*}$  for a given task or environment can be decomposed into individual utility functions which only condition on the individual actions  $a_{i}$ . This assumption results in a space of MARL tasks where decentralized policies (i.e. greedy policies over  $Q_{i}$ ) can be learned to collectively act optimally. The decomposed utility functions  $Q_{i}$  can then be used for greedy action selection by individual agents [36].
It is easy to see that under the IGM assumption in Definition A.1, we cannot learn a set of individual utility functions  $\{Q_i\}$  which can accurately represent the optimal Q-functions of many tasks with multiple global optima, including the XOR game. In fact, [5] showed formally that any algorithm under the IGM constraint cannot represent the underlying optimal Q-function in the XOR game.
B Proofs for AlberDICE
B.1 Proof of Proposition 3.1
Proof. Let
$$
L (\nu_ {i}, w _ {i}) := \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \Big [ w _ {i} (s, a _ {i}) \Big (\hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - \alpha \log w _ {i} (s, a _ {i}) \Big) \Big ] + (1 - \gamma) \hat {\mathbf {E}} _ {s _ {0} \in D _ {0}} [ \nu_ {i} (s _ {0}) ]
$$
Note that  $L(\nu_i, w_i)$  is differentiable and strictly convex for  $w_i$ . Therefore, we only need to find a point where the gradient becomes zero. For any  $x \in D_{\rho_i}$ ,
$$
\frac {\partial L \left(\nu_ {i} , w _ {i}\right)}{\partial w _ {i} (x)} = \hat {e} _ {\nu_ {i}} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) - \alpha \log w _ {i} (x) - \alpha = 0 \tag {18}
$$
$$
\Longleftrightarrow \hat {w} _ {i} ^ {*} (x) = \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) - 1\right) \tag {19}
$$
B.2 Convexity of  $L(\nu_{i})$
Proposition B.1. Let  $L(\nu_i)$  be a function, defined as:
$$
L \left(\nu_ {i}\right) := \bar {\rho} _ {i} \alpha \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} \left[ \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) - 1\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} \left[ \nu_ {i} \left(s _ {0}\right) \right] \tag {20}
$$
Then,  $L(\nu_i)$  is convex with respect to  $\nu_{i}$
Proof. For any functions  $\nu_{i},\nu_{i}^{\prime}$  and constant  $\lambda \in [0,1]$ , we can derive the following equality:
$$
\begin{array}{l} \hat {e} _ {\left(\lambda \nu_ {i} + (1 - \lambda) \nu_ {i} ^ {\prime}\right)} \left(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}\right) \\ = r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \log \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} + \gamma (\lambda \nu_ {i} + (1 - \lambda) \nu_ {i} ^ {\prime}) (s ^ {\prime}) - (\lambda \nu_ {i} + (1 - \lambda) \nu_ {i} ^ {\prime}) (s) \\ = r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \log \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} + \gamma \lambda \nu_ {i} (s ^ {\prime}) - \lambda \nu_ {i} (s) + \gamma (1 - \lambda) \nu_ {i} ^ {\prime} (s ^ {\prime}) - (1 - \lambda) \nu_ {i} ^ {\prime} (s) \\ = \lambda r (s, a _ {i}, \mathbf {a} _ {- i}) - \lambda \alpha \log \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} + \gamma \lambda \nu_ {i} (s ^ {\prime}) - \lambda \nu_ {i} (s) \\ + (1 - \lambda) r (s, a _ {i}, \mathbf {a} _ {- i}) - (1 - \lambda) \alpha \log \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s , a _ {i})} + \gamma (1 - \lambda) \nu_ {i} ^ {\prime} (s ^ {\prime}) - (1 - \lambda) \nu_ {i} ^ {\prime} (s) \\ = \lambda \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) + (1 - \lambda) \hat {e} _ {\nu_ {i} ^ {\prime}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}). \\ \end{array}
$$
Thus,  $e_{\nu_i}$  is the linear function with respect to  $\nu_{i}$ . Furthermore, using the convexity of  $\exp (\cdot)$ ,
$$
\begin{array}{l} \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {D}} \left[ \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s)} \exp \left(\hat {e} _ {\left(\lambda \nu_ {i} + (1 - \lambda) \nu_ {i} ^ {\prime}\right)} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ = \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {D}} \left[ \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s)} \exp \left(\lambda \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) + (1 - \lambda) \hat {e} _ {\nu_ {i} ^ {\prime}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ \leq \lambda \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {D}} \left[ \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s)} \exp \left(\hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ + (1 - \lambda) \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {D}} \left[ \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s)} \exp \left(\hat {e} _ {\nu_ {i} ^ {\prime}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime})\right) \right] \\ \end{array}
$$
Therefore,  $L(\nu_{i})$  is convex function with respect to  $\nu_{i}$
We also show that our practical algorithm minimizes the upper bound of the original optimization problem in equation (7) restated here:
$$
L \left(\nu_ {i}, w _ {i}\right) := (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} \left[ \nu_ {i} \left(s _ {0}\right) \right] + \mathbb {E} _ {\left(s, a _ {i}\right) \sim d ^ {D}} \left[ w _ {i} \left(s, a _ {i}\right) \left(e _ {\nu_ {i}} \left(s, a _ {i}\right) - \alpha \log w _ {i} \left(s, a _ {i}\right)\right) \right] \tag {21}
$$
Corollary B.2.  $L(\nu_i)$  is an upper bound of  $L(\nu_i, w_i^*)$ , where  $w_i^* = \arg \max_{w_i} L(\nu_i, w_i)$ , i.e.,  $L(\nu_i, w_i^*) \leq L(\nu_i)$  always holds, where equality holds when the MDP transition and other agents' policy  $\pi_{-i}$  are deterministic.
Proof. We first note that the closed-form solution for  $\arg \max_{w_i} L(\nu_i, w_i)$  is as follows:
$$
w _ {i} ^ {*} (s, a _ {i}) = \exp \left(\frac {1}{\alpha} e _ {\nu_ {i}} (s, a _ {i}) - 1\right). \tag {22}
$$
By plugging this into equation (7), we obtain
$$
\min  _ {\nu_ {i}} \alpha \mathbb {E} _ {(s, a _ {i}) \sim d ^ {D}} \left[ \exp \left(\frac {1}{\alpha} e _ {\nu_ {i}} (s, a _ {i}) - 1\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] =: L \left(w _ {i} ^ {*}, \nu_ {i}\right)
$$
From Jensen's inequality, we get
$$
\begin{array}{l} L \left(w _ {i} ^ {*}, \nu_ {i}\right) = \alpha \mathbb {E} _ {\left(s, a _ {i}\right) \sim d ^ {D}} \left[ \exp \left(\frac {1}{\alpha} e _ {\nu_ {i}} (s, a _ {i}) - 1\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} \left[ \nu_ {i} \left(s _ {0}\right) \right] \\ = \alpha \mathbb {E} _ {(s, a _ {i}) \sim d ^ {D}} \Big [ \exp \left(\frac {1}{\alpha} \mathbb {E} _ {\substack {\mathbf {a} _ {- i} \sim \pi_ {- i} \\ s ^ {\prime} \sim P (s, a _ {i}, \mathbf {a} _ {- i})}} \big [ \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) ] - 1\right) \Big ] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} \big [ \nu_ {i} (s _ {0}) \big ] \\ \leq \alpha \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {D}} \left[ \frac {\pi_ {- i} (\mathbf {a} _ {- i} | s)}{\pi_ {- i} ^ {D} (\mathbf {a} _ {- i} | s)} \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) - 1\right) \right] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] \\ = L \left(\nu_ {i}\right) \tag {23} \\ \end{array}
$$
Also, the inequality becomes tight when the transition model and the opponent policies are deterministic, since  $\exp \left(\frac{1}{\alpha}\mathbb{E}_{\substack{s' \sim P(s,a_i,\mathbf{a}_{-i}) \\ \hat{e} (s,a_i,\mathbf{a}_{-i},s')}}\right) = \mathbb{E}_{\substack{\mathbf{a}_{-i} \sim \pi_{-i} \\ s' \sim P(s,a_i,\mathbf{a}_{-i})}}\left[\exp \left(\frac{1}{\alpha}\hat{e} (s,a_i,a_{-i},s')\right)\right]$
should always hold for the deterministic transition  $P$  and opponent policies  $\pi_{-i}$ .
C Problems with Naive Extension of OptiDICE to Offline MARL
OptiDICE [14] is a (single-agent) offline policy optimization algorithm which is derived from the regularized Linear Programming (LP) formulation for RL. For a given MDP  $\langle S, A, P, R, \gamma \rangle$ , the derivation of OptiDICE starts with the regularized dual of the LP:
$$
\max  _ {d \geq 0} \sum_ {s, a} d (s, a) r (s, a) - \alpha \mathrm {D} _ {\mathrm {K L}} (d \| d ^ {D}) \tag {24}
$$
$$
\text {s . t .} \sum_ {a ^ {\prime}} d \left(s ^ {\prime}, a ^ {\prime}\right) = (1 - \gamma) p _ {0} \left(s ^ {\prime}\right) + \gamma \sum_ {s, a} P \left(s ^ {\prime} \mid s, a\right) d (s, a), \forall s ^ {\prime}. \tag {25}
$$
Here,  $d(s,a)$  should be a stationary distribution of some policy  $\pi$  by the Bellman flow constraints (25)  $(d^{\pi}(s,a)\coloneqq (1 - \gamma)\sum_{t = 0}^{\infty}\gamma^{t}\operatorname *{Pr}(s_{t} = s,a_{t} = a;\pi))$ , and  $d^{D}$  is the dataset distribution. The goal is to maximize the rewards while not deviating too much from the data distribution, following the conservatism principle in offline RL. Without the regularization term  $\mathrm{D}_{\mathrm{KL}}(d||d^{D})$ , the optimal solution of (24-25) is the stationary distribution for the optimal policy,  $d^{*} = d^{\pi^{*}}$ .
One of the main contributions of OptiDICE is a tractable re-formulation of the dual LP problem above into a single convex optimization problem,
$$
\min  _ {\nu} (1 - \gamma) \mathbb {E} _ {s \sim \mu_ {0}} [ \nu (s) ] + \mathbb {E} _ {(s, a) \sim d ^ {D}} \left[ w _ {\nu} ^ {*} (s, a) e _ {\nu} (s, a) - \alpha w _ {\nu} ^ {*} (s, a) \log w _ {\nu} ^ {*} (s, a) \right]. \tag {26}
$$
$$
\text {w h e r e} w _ {\nu} ^ {*} (s, a) := \exp \left(\frac {1}{\alpha} \left(r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} [ \nu (s ^ {\prime}) ] - \nu (s)\right) - 1\right). \tag {27}
$$
Here,  $\nu(s) \in \mathbb{R}$  is the Lagrangian multiplier for the Bellman flow constraint (25), and it approaches the optimal state value function  $V^{*}(s)$  as  $\alpha \to 0$ . Once we obtain the optimal solution of (26),  $\nu^{*}$ , it was shown that the stationary distribution corrections of the optimal policy is given by  $w_{\nu^{*}}^{*}$ :
$$
w _ {\nu^ {*}} ^ {*} (s, a) = \exp \left(\frac {1}{\alpha} \left(r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \left[ \nu^ {*} \left(s ^ {\prime}\right) \right] - \nu^ {*} (s)\right) - 1\right) = \frac {d ^ {*} (s , a)}{d ^ {D} (s , a)}. \tag {28}
$$
However, its extension to MARL can cause a number of subtle issues. While solving (26) does not suffer from the curse of dimensionality posed in MARL since  $\nu(s)$  is a state-dependent function,  $\nu^{*}$  itself is not an executable policy. We therefore should extract a policy from it. However, once we try to learn a parametric function for  $w(s, a)$ , we encounter a combinatorial space of joint actions. We thus should avoid learning any state-action dependent functions for policy extraction. One feasible way to do so is to perform policy extraction via Weighted Behavior-cloning (WBC):
$$
\forall i, \max  _ {\pi_ {i}} \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {D}} \left[ \hat {w} _ {\nu^ {*}} ^ {*} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \log \pi_ {i} (a _ {i} | s) \right] \approx \mathbb {E} _ {(s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) \sim d ^ {*}} [ \log \pi_ {i} (a _ {i} | s) ] \tag {29}
$$
$$
\text {s . t .} \hat {w} _ {\nu^ {*}} ^ {*} (s, a _ {i}, \mathbf {a} _ {- i}, s ^ {\prime}) = \exp \left(\frac {1}{\alpha} \left(r (s, a _ {i}, \mathbf {a} _ {- i}) + \gamma \nu^ {*} (s ^ {\prime}) - \nu^ {*} (s)\right) - 1\right), \tag {30}
$$
which corresponds to behavior-cloning of the factorized policy on the state-action visits by the optimal (joint) policy. However, if the optimal joint policy (by  $\nu^{*}$ ) is a multi-modal distribution, this WBC policy extraction step can result in an arbitrarily bad policy, selecting OOD joint actions. For example, consider the XOR matrix game in Figure 1 with a dataset  $D = \{(A,A),(A,B),(B,A)\}$ , where the optimal joint policy is given by  $\pi^{*}(a_{1} = A,a_{2} = B) = \pi^{*}(a_{1} = A,a_{2} = B) = \frac{1}{2}$ . In this situation, WBC of OptiDICE (29) obtains the factorized policies of  $\pi_1(a_1 = A) = \pi_1(a_1 = B) = \frac{1}{2}$  and  $\pi_2(a_2 = A) = \pi_2(a_2 = B) = \frac{1}{2}$ , which can select suboptimal (and OOD) joint actions:
$$
\pi \left(a _ {1} = A, a _ {2} = A\right) = \pi_ {1} \left(a _ {1} = A\right) \pi_ {2} \left(a _ {2} = A\right) = \frac {1}{4}
$$
$$
\pi \left(a _ {1} = A, a _ {2} = B\right) = \pi_ {1} \left(a _ {1} = A\right) \pi_ {2} \left(a _ {2} = B\right) = \frac {1}{4}
$$
$$
\pi \left(a _ {1} = B, a _ {2} = A\right) = \pi_ {1} \left(a _ {1} = B\right) \pi_ {2} \left(a _ {2} = A\right) = \frac {1}{4}
$$
$$
\pi \left(a _ {1} = B, a _ {2} = B\right) = \pi_ {1} \left(a _ {1} = B\right) \pi_ {2} \left(a _ {2} = B\right) = \frac {1}{4}
$$
This analysis is consistent with our experimental results in Section 5, which demonstrated the failure of OptiDICE in solving the Penalty XOR Game as well as Bridge by converging to sub-optimal OOD joint actions.
D Proofs for Section 4
To prove Theorem 4.2, we first need to show that the regularized objective in the LP formulation of AlberDICE, cf. equation (2), preserves the common reward structure of  $G$ .
Proof. Recall that the KL-divergence,  $\mathrm{D}_{\mathrm{KL}}(p(x)\| q(x))$ , between probability distributions  $p$  and  $q$  is defined as  $\mathrm{D}_{\mathrm{KL}}(p(x)\| q(x)) := \sum_{x} p(x) \log \frac{p(x)}{q(x)}$ . Thus, the  $\mathrm{D}_{\mathrm{KL}}$  term in the objective of equation (2) can be written as
$$
\mathrm {D} _ {\mathrm {K L}} \left(d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \| d ^ {D} (s, a _ {i}, \mathbf {a} _ {- i})\right) = \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \cdot \log \frac {d _ {i} ^ {\pi} (s , a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s)}{d ^ {D} (s , a _ {i} , \mathbf {a} _ {- i})}.
$$
Since the  $\pi_i^{\prime}s$  are factorized by assumption, the decomposition  $d_{i}^{\pi}(s,a_{i}) = d^{\pi}(s)\pi_{i}(a_{i}|s)$ , implies that the numerator in the log-term of the previous expression can be written as
$$
d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) = d ^ {\pi} (s) \pi_ {i} (a _ {i} | s) \pi_ {- i} (\mathbf {a} _ {- i} | s) = d ^ {\pi} (s) \pi (\mathbf {a} | s).
$$
Substituting back in the initial expression of the objective function, we obtain that
$$
\begin{array}{l} \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \mathrm {D} _ {\mathrm {K L}} \left(d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \| d ^ {D} (s, a _ {i}, \mathbf {a} _ {- i})\right) \\ = \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \cdot \log \frac {d ^ {\pi} (s) \pi (\mathbf {a} | s)}{d ^ {D} (s , a _ {i} , \mathbf {a} _ {- i})} \\ = \sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \left[ r (s, a _ {i}, \mathbf {a} _ {- i}) - \alpha \log \frac {d ^ {\pi} (s) \pi (\mathbf {a} | s)}{d ^ {D} (s , a _ {i} , \mathbf {a} _ {- i})} \right]. \\ \end{array}
$$
Thus, by setting  $\tilde{r}(s,a_i,\mathbf{a}_{-i}) := r(s,a_i,\mathbf{a}_{-i}) - \alpha \cdot \log \frac{d^\pi(s)\pi(\mathbf{a}|s)}{d^D(s,a_i,\mathbf{a}_{-i})}$ , for all  $(s,\mathbf{a}) \in S \times \mathcal{A}$ , we obtain the claim. The equality of the last expression for all  $i \in \mathcal{N}$  follows now immediately from application of the decomposition  $d_i^\pi (s,a_i)\pi_{-i}(\mathbf{a}_{-i}|d) = d^\pi (s)\pi (\mathbf{a}|s)$ , on the outer expectation for all agent  $i \in \mathcal{N}$ .