SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The optimal dataset (500 trajectories) was constructed by a hand-crafted (multi-modal) optimal policy which randomizes between Agent 1 crossing the bridge first while Agent 2 retreats, and vice-versa. The mix dataset is a mixture between 500 trajectories from the optimal dataset and 500 trajectories generated by a uniform random policy.
F.2 Multi-Robot Warehouse (RWARE)
For the data collection policy used to construct the dataset, we train Multi-Agent Transformers (MAT) [37] which takes an autoregressive policy structure and thus is able to generate diverse behavior. We further train MAT over 3 random seeds, and generate a expert dataset with a mixture of diverse behaviors.
F.3 Google Research Football
Similar to the dataset collection procedure in RWARE, we use MAT to generate a medium-expert dataset in order to ensure that agents score goals in different ways. Similar to Bridge, we construct a dataset of 2000 trajectories where 1000 trajectories have medium performance (roughly  $60\%$  performance of the expert policies) and another 1000 from fully trained "expert" MAT policies.
F.4 SMAC
We use the public dataset provided by [31].
G Matrix Game Results
In Table 6, we show results for the converged policies in the Matrix Game presented in Section 5 of the main text and shown again in Figure 6.
Figure 6: XOR Game with Penalty
As expected, all algorithms converge to the optimal action  $AB$  for the dataset  $D_{(a)}$ . However, AlberDICE is the only algorithm which can choose the optimal action deterministically for all 4 datasets, showing robustness even when the environment has multiple global optima and the dataset is generated by a mixture of diverse policies. Here we can consider any dataset containing both  $AB$  and  $BA$  as a mixture of diverse data collection policies where the two agents cooperate to select the optimal actions but in different ways. For OMAR, we see that it is able to learn the optimal policy for  $D_{(b)}$ . However, as discussed in the Introduction of the main text, OMAR degenerates to the sub-optimal action  $BB$  for dataset  $D_{(c)}$  because each agent acts independently and assumes the other agent chooses the individual action  $A$  with a  $\frac{2}{3}$  probability. Finally, we also note that BC can fail and converge to OOD joint actions even in  $D_{(b)}$  where the dataset is optimal.
Table 6: Policy values after convergence for the Matrix Game in Figure 2. The policy values are calculated by multiplying the individual policy values for each agent i.e.  $\pi = \pi_1 \times \pi_2$ . The datasets consist of  $D_{(a)} = \{AB\}$ ,  $D_{(b)} = \{AB, BA\}$ ,  $D_{(c)} = \{AA, AB, BA\}$ ,  $D_{(d)} = \{AA, AB, BA, BB\}$ .
H Bridge Policy Visualizations
The visualizations for learned policies of AlberDICE, OptiDICE, ICQ, OMAR, BC, and MADTKD are shown for all state possibilities. From Figure 7, it is clear that AlberDICE is the only algorithm which reliably chooses a deterministic action (Left, Left) at the initial state. However, since the visualizations are provided for the optimal dataset which has a small coverage of states, the policy values may still be sub-optimal at OOD states. The results for OMAR in Figure 10 shows that the agents are acting independently without regard for the other agents, and converges to (Left, Right) at the initial state which results in a collision. Finally, we note that the visualizations for MADTKD may not exactly correspond to the policy values used in the experimental results in Section 5. This is because MADTKD uses a history-dependent Transformer policy and it is not clear how to visualize the policy values depending on the history. The visualizations shown in Figure 11 assume that each agent is at the given state in the initial timestep, which is different from encountering that state after many timesteps. Nonetheless, the results for the true initial state shown in the central portions of the figure are consistent with the quantitative results in Section 5.
Figure 7: Policy Visualizations for AlberDICE on the Bridge (Hard) environment for the optimal dataset where the arrows show the probability of choosing a particular action given that the other agents are in  $\bullet$  and  $\bullet$  for agents 1 and 2, respectively.
Figure 8: Policy Visualizations for OptiDICE on the Bridge (Hard) environment for the optimal dataset where the arrows show the probability of choosing a particular action given that the other agents are in  $\bullet$  and  $\bullet$  for agents 1 and 2, respectively.
Figure 9: Policy Visualizations for ICQ on the Bridge (Hard) environment for the optimal dataset where the arrows show the probability of choosing a particular action given that the other agents are in  $\bullet$  and  $\bullet$  for agents 1 and 2, respectively.
Figure 10: Policy Visualizations for OMAR on the Bridge (Hard) environment for the optimal dataset where the arrows show the probability of choosing a particular action given that the other agents are in  $\bullet$  and  $\bullet$  for agents 1 and 2, respectively.
Figure 11: Policy Visualizations for MADTKD on the Bridge (Hard) environment for the optimal dataset where the arrows show the probability of choosing a particular action given that the other agents are in  $\bullet$  and  $\bullet$  for agents 1 and 2, respectively.
Figure 12: Policy Visualizations for BC on the Bridge (Hard) environment for the optimal dataset where the arrows show the probability of choosing a particular action given that the other agents are in  $\bullet$  and  $\bullet$  for agents 1 and 2, respectively.
I Additional Ablation Results
Here we show ablation results for hyperparameter  $\alpha$  which controls the degree of conservatism. Table 7 shows that AlberDICE is not too sensitive to the hyperparameter  $\alpha$  as long as it is within a reasonable range.
α	0.001	0.01	0.1	1	10	100	1000
CA-Hard (GRF) (N=4)	0.00 ± 0.00	0.08 ± 0.11	0.82 ± 0.03	0.84 ± 0.06	0.78 ± 0.09	0.78 ± 0.11	0.79 ± 0.11
Corridor (SMAC) (N=8)	0.00 ± 0.00	0.00 ± 0.00	0.92 ± 0.02	0.97 ± 0.01	0.95 ± 0.03	0.92 ± 0.03	0.91 ± 0.04
Table 7: Ablation Study for Conservatism Hyperparameter  $\alpha$  (over 3 random seeds)
J Implementation Details
J.1 AlberDICE for Dec-POMDP
While our paper focuses on MMDPs, our experimental results show that we can extend AlberDICE to Dec-POMDPs  $G = \langle \mathcal{N},\mathcal{S},\mathcal{A},r,P,p_0,\gamma ,\Omega ,O\rangle$  by using partial observations and a history-dependent policy in place of a state-dependent policy for each agent. In Dec-POMDP, each agent  $i$  observes individual observations  $o_{i}\in \Omega$  which is given by the observation function  $O:S\times \mathcal{A}\to \Omega$ . Each agent makes decision based on the observation-action history  $\tau_{i}\in (\Omega \times A)^{t - 1}\times \Omega$ , where each agent's (decentralized) policy is represented as  $\pi_i(a_i|\tau_i)$ .
For the Matrix Game, Bridge and GRF, we used an MLP policy since the partial observations for each agent correspond with the global state (i.e., each individual policy is conditioned only on its current observation  $o_i$ , rather than the entire history). For Warehouse, each individual policy uses the partial observations as input, which is the 3x3 neighborhood surrounding each agent. We utilize a Transformer-based policy in order for the agent to condition on the history of local observations and actions, while speeding up training in comparison to Recurrent Neural Networks (RNNs). This same Transformer-based policy is used for all baselines as well.
During centralized training, AlberDICE uses the global state  $s$  for training  $\pi_{\beta_{-i}}^{D}(\mathbf{a}_{-i}|s,a_{i}),\nu_{\theta_{i}}(s)$  and  $e_{\psi_i}(s,a)$  for all environments, where its training procedure is identical to the MMDP training procedure described in Appendix E.3. Only the policy extraction step is different for Dec-POMDP, where each agent's history-dependent policies are trained by:
$$
\min  _ {\beta_ {i}} - \hat {\mathbf {E}} _ {(\tau_ {i}, a _ {i}) \in D} \left[ \log \pi_ {\beta_ {i}} ^ {D} (a _ {i} \mid \tau_ {i}) \right] \tag {49}
$$
$$
\min  _ {\phi_ {i}} \hat {\mathbf {E}} _ {(s, a _ {i}, \tau_ {i}) \in D} \left[ \sum_ {a _ {i}} \pi_ {\phi_ {i}} (a _ {i} | \tau_ {i}) \left(- e _ {\psi_ {i}} (s, a _ {i}) + \alpha \log \frac {\pi_ {\phi_ {i}} \left(a _ {i} \mid \tau_ {i}\right)}{\pi_ {\beta_ {i}} ^ {D} \left(a _ {i} \mid \tau_ {i}\right)}\right) \right] \tag {50}
$$
J.2 Hyperparameter Details
We conduct minimal hyperparameter tuning for all algorithms for fair comparisons. It is also worth noting that in offline RL, it is important to develop algorithms which require minimal hyperparameter tuning [23]. We chose the best values for  $\alpha$  for both AlberDICE and OptiDICE between  $[0.1, 1, 5, 10]$  on  $\mathrm{N} = 2$  (Tiny) for RWARE and RPS (2 agents) for GRF. The best values were then used for all scenarios thereafter. The final values used were  $\alpha = 0.1$  (GRF) and  $\alpha = 1$  (RWARE) for AlberDICE and  $\alpha = 1$  (GRF, RWARE) for OptiDICE. We found that the performance gaps between different hyperparameters were minimal as long as they were within a reasonable range where training is numerically stable.
For ICQ [40], we found that the algorithm tends to become numerically unstable after a certain number of training epochs even with sufficient hyperparameter tuning due to the exploding Q values, especially in the GRF and SMAC environment.
K Computational Resources
For Warehouse experiments, we utilized a single NVIDIA Geforce RTX 3090 graphics processing unit (GPU). The experiments for running AlberDICE took  $5\mathrm{H}$ ,  $14\mathrm{H}$ , and  $29\mathrm{H}$  for 2, 4, 6 agent environments, respectively.