ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

## Abstract

Abstract Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift , which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents’ local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi-agent value decomposition strategy to facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.

## 1 Introduction

Over the years, deep RL has achieved remarkable success in various decision-making tasks (Levine et al., ; Silver et al., ; Kalashnikov et al., ; Haydari & Yılmaz, ). However, a significant limitation of deep RL is its need for millions of interactions with the environment to gather experiences for policy improvement. This process can be both costly and risky, especially in real-world applications like robotics and healthcare.
To address this challenge, offline RL has emerged, enabling policy learning based solely on pre-collected demonstrations (Levine et al., ). Despite this advancement, offline RL faces a critical issue: the distribution shift between the offline dataset and the learned policy (Kumar et al., ). This distribution shift complicates value estimation for unseen states and actions during policy evaluation, resulting in extrapolation errors where out-of-distribution (OOD) state-action pairs are assigned unrealistic values (Fujimoto et al., ).

To tackle OOD actions, many existing works impose action-level constraints, either implicitly by regulating the learned value functions or explicitly through distance or divergence penalties (Fujimoto et al., ; Kumar et al., ; Wu et al., ; Peng et al., ; Fujimoto & Gu, ; Xu et al., ). Only a few recent studies have addressed both OOD actions and states using state-action-level behavior constraints (Li et al., ; Zhang et al., ; Lee et al., ; ; Mao et al., ). In particular, there is an important line of work on DIstribution Correction Estimation (DICE) (Nachum & Dai, ) that constrains the distance in terms of the joint state-action occupancy measure between the learning policy and the offline policy. These DICE-based methods have demonstrated impressive performance results on the D4RL benchmarks (Lee et al., ; ; Mao et al., ).

It is important to note that that all the aforementioned offline RL approaches primarily focus on the single-agent setting. While multi-agent setting is prevalent in many real-world sequential decision-making tasks, offline MARL remains a relatively under-explored area. The multi-agent setting poses significantly greater challenges due to the large joint state-action space, which expands exponentially with the number of agents, as well as the inter-dependencies among the local policies of different agents. As a result, the offline data distribution can become quite sparse in these high-dimensional joint action spaces, leading to an increased number of OOD state-action pairs and exacerbating extrapolation errors.
A few recent studies have sought to address the negative effects of sparse data distribution in offline MARL by adapting the well-known centralized training decentralized execution (CTDE) paradigm from online MARL (Oliehoek et al., ; Kraemer & Banerjee, ), enabling data-related regularization at the individual agent level. Notably, some of these works (Pan et al., ; Shao et al., ; Wang et al., ) extend popular offline single-agent RL algorithms, such as CQL (Kumar et al., ) and SQL/EQL (Xu et al., ), within the CTDE framework.

In our work, we focus on addressing the aforementioned challenges in offline cooperative MARL. In particular, we follow the DICE approach to address both OOD states and actions, motivated by remarkable performance of recent DICE-based methods in offline single-agent RL. Similar to previous works in offline MARL, we adopt the CTDE framework to handle exponential joint state-action spaces in the multi-agent setting. We remark that extending the DICE approach under this CTDE framework is not straightforward given the complex objective of DICE that involves the f-divergence in stationary distribution between the learning joint policy and the behavior policy. Therefore, the value decomposition in CTDE needs to be carefully designed to ensure the consistency in optimality between the global and local policies. In particular, we provide the following main contributions:

- •
We propose ComaDICE, a new offline MARL algorithm that integrates DICE with a carefully designed value decomposition strategy. In ComaDICE, under the CTDE framework, we decompose both the global value function $\nu^{tot}$ and the global advantage functions $A^{tot}_{\nu}$, rather than using Q-functions as in previous MARL works.
This unique factorization approach allows us to theoretically demonstrate that the global learning objective in DICE is convex in local values, provided that the mixing network used in the value decomposition employs non-negative weights and convex activation functions. This significant finding ensures that our decomposition strategy promotes an efficient and stable training process.
- •
Building on our decomposition strategy, we demonstrate that finding an optimal global policy can be divided into multiple sub-problems, each aims to identify a local optimal policy for an individual agent. We provide a theoretical proof that the global optimal policy is, in fact, equivalent to the product of the local policies derived from these sub-problems.
- •
Finally, we conduct extensive experiments to evaluate the performance of our algorithm, ComaDICE, in complex MARL environments, including: multi-agent StarCraft II (i.e., SMACv1 (Samvelyan et al., ), SMACv2 (Ellis et al., )) and multi-agent Mujoco (de Witt et al., ) benchmarks. Our empirical results show that our ComaDICE outperforms several strong baselines in all these benchmarks.

## 2 Related Work

##### Offline Reinforcement Learning (offline RL).

Offline RL focuses on learning policies from pre-collected datasets without any further interactions with the environment (Levine et al., ; Prudencio et al., ). A significant challenge in offline RL is the issue of distribution shift, where unseen actions and states may arise during training and execution, leading to inaccurate policy evaluations and suboptimal outcomes. Consequently, there is a substantial body of literature addressing this challenge through various approaches (Prudencio et al., ).
In particular, some studies impose explicit or implicit policy constraints to ensure that the learned policy remains close to the behavioral policy (Fujimoto et al., ; Kumar et al., ; Wu et al., ; Kostrikov et al., ; Peng et al., ; Nair et al., ; Fujimoto & Gu, ; Xu et al., ; Cheng et al., ; Li et al., ). Others incorporate regularization terms into the learning objectives to mitigate the value overestimation on OOD actions (Kumar et al., ; Kostrikov et al., ; Xu et al., ; Niu et al., ; Xu et al., ; Wang et al., ).
Uncertainty-based offline RL methods seek to balance conservative approaches with naive off-policy RL techniques, relying on estimates of model, value, or policy uncertainty (Agarwal et al., ; An et al., ; Bai et al., ). Offline model-based algorithms focus on conservatively estimating the transition dynamics and reward functions based on the pre-collected datasets (Kidambi et al., ; Yu et al., ; Matsushima et al., ; Yu et al., ). Some other methods impose action-level regularization through imitation learning techniques  (Xu et al., ; Chen et al., ; Zhang et al., ; Zheng et al., ; Brandfonbrener et al., ; Xu et al., ). Finally, while a majority of previous works target OOD actions only, there are a few recent works attempt to address both OOD states and actions (Li et al., ; Zhang et al., ; Lee et al., ; ; Sikchi et al., ; Mao et al., ). Our work on offline MARL follow the DICE-based approach, as motivated by compelling performance of DICE-based algorithms in single-agent settings (Lee et al., ; ; Sikchi et al., ; Mao et al., ).

##### Offline Multi-agent Reinforcement Learning (offline MARL).

While there is a substantial body of literature on offline single-agent RL, research on offline MARL remains limited. Offline MARL faces challenges from both distribution shift—characteristic of offline settings—and the exponentially large joint action space typical of multi-agent environments. Recent studies have begun to merge advanced methodologies from both offline RL and MARL to address these challenges (Yang et al., ; Pan et al., ; Shao et al., ; Wang et al., )
Specifically, these works employ local policy regularization within the centralized training with decentralized execution (CTDE) framework to mitigate distribution shift. The CTDE paradigm, well-established in online MARL, facilitates more efficient and stable learning while allowing agents to operate in a decentralized manner (Oliehoek et al., ; Kraemer & Banerjee, ). For instance, Yang et al. () utilize importance sampling to manage local policy learning on OOD samples. Both works by Pan et al. () and Shao et al. () are built upon CQL (Kumar et al., ), a prominent offline RL algorithm for single-agent scenarios. Matsunaga et al. () adopt the Nash equilibrium solution concept in game theory to iteratively update best responses of individual agents. Finally, OMIGA (Wang et al., ) establishes the equivalence between global and local value regularization within a policy constraint framework, making it the current state-of-the-art algorithm in offline MARL.

Beyond this main line of research, some studies formulate offline MARL as a sequence modeling problem, employing supervised learning techniques to tackle the issue (Meng et al., ; Tseng et al., ), while others adhere to decentralized approaches (Jiang & Lu, ).

## 3 Preliminaries

Our work focuses on cooperative multi-agent RL, which can be modeled as a multi-agent Partially Observable Markov Decision Process (POMDP), defined by the tuple ${\mathcal{M}}=\langle{\mathcal{S}},{\mathcal{A}},P,r,{\mathcal{Z}},{\mathcal{O
}},n,{\mathcal{N}},\gamma\rangle$. Here, $n$ is number of agents, ${\mathcal{N}}=\{1,\ldots,n\}$ is the set of agents, $\textbf{s}\in S$ represents the true state of the multi-agent environment, and ${\mathcal{A}}=\prod_{i\in{\mathcal{N}}}{\mathcal{A}}_{i}$ is the set of joint actions, where ${\mathcal{A}}_{i}$ is the set of individual actions available to agent $i\in{\mathcal{N}}$. At each time step, each agent $i\in\{1,2,\dots,n\}$ selects an action $a_{i}\in{\mathcal{A}}_{i}$, forming a joint action $\textbf{a}=(a_{1},a_{2},\dots,a_{n})\in{\mathcal{A}}$. The transition dynamics $P(\textbf{s}^{\prime}|\textbf{s},\textbf{a}):{\mathcal{S}}\times{\mathcal{A}}
\times{\mathcal{S}}\to[0,1]$ describe the probability of transitioning to the next state $\textbf{s}^{\prime}$ when agents take an action a from the current state s. The discount factor $\gamma\in[0,1)$ represents the weight given to future rewards. In a partially observable environment, each agent receives a local observation $o_{i}\in{\mathcal{O}}_{i}$ based on the observation function ${\mathcal{Z}}_{i}(\textbf{s}):{\mathcal{S}}\to{\mathcal{O}}_{i}$, and we denote the joint observation as $\textbf{o}=(o_{1},o_{2},\dots,o_{n})$. In cooperative MARL, all agents share a global reward function $r(\textbf{o},\textbf{a}):{\mathcal{O}}\times{\mathcal{A}}\to\mathbb{R}$. The goal of all agents is to learn a joint policy $\boldsymbol{\pi}_{\text{tot}}=\{\pi_{1},\dots,\pi_{n}\}$ that collectively maximize the expected discounted returns $\mathbb{E}_{(\textbf{o},\textbf{a})\sim\boldsymbol{\pi}_{\text{tot}}}\left[
\sum_{t=0}^{\infty}\gamma^{t}r(\textbf{o}_{t},\textbf{a}_{t})\right]$. In the offline MARL setting, a pre-collected dataset ${\mathcal{D}}$ is obtained by sampling from a behavior policy $\mu_{\text{tot}}=\{\mu_{1},\dots,\mu_{n}\}$, and the policy learning is conducted soly based on ${\mathcal{D}}$, with no interactions with the environment. We also define the occupancy measure (or stationary distribution) as follows:

$$ $\rho^{\boldsymbol{\pi}_{tot}}(\textbf{o},\textbf{a})=(1-\gamma)\sum\nolimits_{ t=0}^{\infty}P(\textbf{o}_{t}=\textbf{o},~{}\textbf{a}_{t}=\textbf{a})$ $$

which represents distribution visiting the pair (observation, action) $(\textbf{o}_{t},\textbf{a}_{1})$ when following the joint policy $\boldsymbol{\pi}_{tot}$, where $\textbf{s}_{0}\sim P_{0},~{}\textbf{a}_{t}\sim\boldsymbol{\pi}_{tot}(\cdot|
\textbf{s}_{t})$ and $\textbf{s}_{t+1}\sim P(\cdot|\textbf{s}_{t},\textbf{a}_{t})$.

## 4 ComaDICE: Offline Cooperative Multi-Agent RL with Stationary Distribution Correction Estimation

We consider an offline cooperative MARL problem where the goal is to optimize the expected discounted joint reward. In this work, we focus on the DICE objective function Nachum & Dai (), which incorporates a stationary distribution regularizer to capture the divergence between the occupancy measures of the learning policy, $\boldsymbol{\pi}_{tot}$, and the behavior policy, $\mu_{tot}$, formulated as follows:

$$ $\displaystyle\max\nolimits_{\boldsymbol{\pi}_{tot}}~{}~{}\mathbb{E}_{(\textbf{ o},\textbf{a})\sim\rho^{\boldsymbol{\pi}_{tot}}}\left[r(\textbf{o},\textbf{a}) \right]-\alpha D^{f}\left(\rho^{\boldsymbol{\pi}_{tot}}\parallel\rho^{\mu_{tot }}\right)$ (1) $$

where
$D^{f}\left(\rho^{\boldsymbol{\pi}_{tot}}\parallel\rho^{\mu_{tot}}\right)=
\mathbb{E}_{(\textbf{o},\textbf{a})\sim\rho^{\boldsymbol{\pi}_{tot}}}\left[f
\left(\frac{\rho^{\boldsymbol{\pi}_{tot}}}{\rho^{\mu_{tot}}}\right)\right]$
is the f-divergence between the stationary distribution $\rho^{\boldsymbol{\pi}_{tot}}$ of the learning policy and $\rho^{\mu_{tot}}$ of the behavior policy. In this work, we consider $f(\cdot)$ to be strictly convex and differentiable. The parameter $\alpha$ controls the trade-off between maximizing the reward and penalizing deviation from the offline dataset’s distribution (i.e., penalizing distributional shift). When $\alpha=0$, the problem becomes the standard offline MARL, where the objective is to find a joint policy that maximizes the expected joint reward. On the other hand, when $\alpha\gg 1$, the problem shifts towards imitation learning, aiming to closely mimic the behavioral policy.

This DICE-based approach offers the advantage of better capturing the system dynamics inherent in the offline data. Such stationary distributions, $\rho^{\boldsymbol{\pi}_{tot}}$ and $\rho^{\mu_{tot}}$, however, are not directly available. We will discuss how to estimate them in the next subsection.

### 4.1 Constrained Optimization in the Stationary Distribution Space

We first formulate the learning problem in Eq. 1 as a constrained optimization on the space of $\rho^{\boldsymbol{\pi}_{tot}}$:

$$ $\displaystyle\max\nolimits_{\rho^{\boldsymbol{\pi}_{tot}}}\quad$ $\displaystyle\mathbb{E}_{(\textbf{o},\textbf{a})\sim\rho^{\boldsymbol{\pi}_{ tot}}}\left[r(\textbf{o},\textbf{a})\right]-\alpha D^{f}\left(\rho^{ \boldsymbol{\pi}_{tot}}\parallel\rho^{\mu_{tot}}\right)$ (2) s.t. $\displaystyle\sum\nolimits_{\textbf{a}^{\prime}}\rho^{\boldsymbol{\pi}_{tot}}( \textbf{s},\textbf{a}^{\prime})=(1-\gamma)p_{0}(\textbf{s})+\gamma\sum \nolimits_{\textbf{a}^{\prime},\textbf{s}^{\prime}}\rho^{\boldsymbol{\pi}_{tot }}(\textbf{s}^{\prime},\textbf{a}^{\prime})P(\textbf{s}|\textbf{a}^{\prime}, \textbf{s}^{\prime}),~{}\forall\textbf{s}\in{\mathcal{S}}$ (3) $$

When $f$ is convex, (2-3) becomes a convex optimization problem, as it involves maximizing a concave objective function subject to linear constraints.
We now consider the Lagrange dual of (2-3):

$$ $\displaystyle{\mathcal{L}}(\nu^{tot},$ $\displaystyle\rho^{\boldsymbol{\pi}_{tot}})=\mathbb{E}_{(\textbf{o},\textbf{a} )\sim\rho^{\boldsymbol{\pi}_{tot}}}\left[r(\textbf{o},\textbf{a})\right]- \alpha\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[f\left( \frac{\rho^{\boldsymbol{\pi}_{tot}}(\textbf{s},\textbf{a})}{\rho^{\mu_{tot}}( \textbf{s},\textbf{a})}\right)\right]$ $\displaystyle-\sum\nolimits_{\textbf{s}}\nu^{tot}(\textbf{s})\Big{(}\sum \nolimits_{\textbf{a}^{\prime}}\rho^{\boldsymbol{\pi}_{tot}}(\textbf{s}, \textbf{a}^{\prime})-(1-\gamma)p_{0}(\textbf{s})-\gamma\sum\nolimits_{\textbf{ a}^{\prime},\textbf{s}^{\prime}}\rho^{\boldsymbol{\pi}_{tot}}(\textbf{s}^{ \prime},\textbf{a}^{\prime})P(\textbf{s}|\textbf{a}^{\prime},\textbf{s}^{ \prime})\Big{)}$ (4) $$

where $\nu^{tot}(\textbf{s})$ is a Lagrange multiplier. Since (2-3) is a convex optimization problem, it is equivalent to the following minimax problem over the spaces of $\nu^{tot}$ and $\rho^{\boldsymbol{\pi}_{tot}}$:

$$ $\min\nolimits_{\nu^{tot}}\max\nolimits_{\rho^{\boldsymbol{\pi}_{tot}}}\left\{{ \mathcal{L}}(\nu^{tot},\rho^{\boldsymbol{\pi}_{tot}})\right\}$ $$

Furthermore, we observe that ${\mathcal{L}}(\nu^{tot},\rho^{\boldsymbol{\pi}_{tot}})$ is linear in $\nu^{tot}$ and concave in $\rho^{\boldsymbol{\pi}_{tot}}$, so the minimax problem has a saddle point, implying:
$\min_{\nu^{tot}}\max_{\rho^{\boldsymbol{\pi}_{tot}}}\left\{{\mathcal{L}}(\nu^{
tot},\rho^{\boldsymbol{\pi}_{tot}})\right\}=\max_{\rho^{\boldsymbol{\pi}_{tot}
}}\min_{\nu^{tot}}\left\{{\mathcal{L}}(\nu^{tot},\rho^{\boldsymbol{\pi}_{tot}}
)\right\}.$
In a manner analogous to the single-agent case, by defining $w^{tot}_{\nu}(\textbf{s},\textbf{a})=\frac{\rho^{\boldsymbol{\pi}_{tot}}(
\textbf{s},\textbf{a})}{\rho^{\mu_{tot}}(\textbf{s},\textbf{a})}$, the Lagrange dual function can be simplified into the more compact form (with detailed derivations are in the appendix):

$$ $\displaystyle{\mathcal{L}}(\nu^{tot},w^{tot})=(1-\gamma)\mathbb{E}_{\textbf{s} \sim p_{0}}[\nu^{tot}(\textbf{s})]+\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho ^{\mu_{tot}}}\left[-\alpha f\left(w^{tot}_{\nu}(\textbf{s},\textbf{a})\right)+ w^{tot}_{\nu}(\textbf{s},\textbf{a})A^{tot}_{\nu}(\textbf{s},\textbf{a})\right]$ $$

where $A^{tot}_{\nu}$ is an “advantage function” defined based on $\nu^{tot}$ as:

$$ $A^{tot}_{\nu}(\textbf{s},\textbf{a})=q^{tot}(\textbf{s},\textbf{a})-\nu^{tot}( \textbf{s})$ (5) $$

with $q^{tot}(\textbf{s},\textbf{a})=r({\mathcal{Z}}(\textbf{s}),\textbf{a})+\gamma
\mathbb{E}_{\textbf{s}^{\prime}\sim P(\cdot|\textbf{s},\textbf{a})}[\nu^{tot}(
\textbf{s}^{\prime})]$. It is important to note that $\nu^{tot}(\textbf{s})$ and $q^{tot}(\textbf{s},\textbf{a})$ can be interpreted as a value function and a Q-function, respectively, arising from the decomposition of the stationary distribution regularizer. We can now write the learning problem as follows:

$$ $\displaystyle\min\nolimits_{\nu^{tot}}\max\nolimits_{w^{tot}\geq 0}~{}~{}\{{ \mathcal{L}}(\nu^{tot},w^{tot})\}$ (6) $$

It can be observed that ${\mathcal{L}}(\nu^{tot},w^{tot})$ is linear in $\nu^{tot}$ and concave in $w^{tot}$, which ensures well-behaved properties in both the $\nu^{tot}$- and $w^{tot}$-spaces. A key feature of the above minimax problem is that the inner maximization problem has a closed-form solution, which greatly simplifies the minimax problem, making it no longer adversarial. We formalize this result as follows:

###### Proposition 4.1 .

The minimax problem in Eq. 6 is equivalent to
$\min_{\nu^{tot}}~{}~{}\big{\{}\widetilde{{\mathcal{L}}}(\nu^{tot})\big{\}}$, where

$$ $\widetilde{{\mathcal{L}}}(\nu^{tot})=(1-\gamma)\mathbb{E}_{\textbf{s}\sim p_{0 }}[\nu^{tot}(\textbf{s})]+\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{ tot}}}\bigg{[}\alpha f^{*}\bigg{(}\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{ \alpha}\bigg{)}\bigg{]}$ $$

Here, $f^{*}$ is convex conjugate of $f$, i.e., $f^{*}(y)=\sup_{t\geq 0}\{ty-f(t)\}$. Moreover, if $\nu^{tot}$ is parameterized by $\theta$, the first order derivative of $\widetilde{{\mathcal{L}}}(\nu^{tot})$ w.r.t. $\theta$ is given as follows:

$$ $\nabla_{\theta}\widetilde{{\mathcal{L}}}(\nu^{tot})=(1-\gamma)\mathbb{E}_{ \textbf{s}\sim p_{0}}[\nabla_{\theta}\nu^{tot}(\textbf{s})]+\mathbb{E}_{( \textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[\nabla_{\theta}A^{tot}_{\nu}( \textbf{s},\textbf{a})w^{tot*}_{\nu}(\textbf{s},\textbf{a})\right]$ $$

where $w^{tot*}_{\nu}(s,a)=\max\{0,{f^{\prime}}^{-1}(A^{tot}_{\nu}(\textbf{s},\textbf
{a})/\alpha)\}$, with $f^{\prime-1}(\cdot)$ is the inverse function of the first-order derivative of $f$.

###### Proposition 4.1 .

### 4.2 Value Factorization

Directly optimizing $\min_{\nu^{tot}}~{}~{}\left\{{\mathcal{L}}(\nu^{tot},w^{tot*}_{\nu})\right\}$ in multi-agent settings is generally impractical due to the large state and action spaces. Therefore, we follow the idea of value decomposition in the well-known CTDE framework in cooperative MARL to address this computational challenge. However, it is not straightforward to extend the DICE approach within this CTDE framework due to the complex objective of DICE, which involves the f-divergence between the learnt joint policy and the behavior policy in stationary distributions. It is thus crucial to carefully design the value decomposition in CTDE to ensure optimality consistency between the global and local policies.

Specifically, we adopt a factorization approach that decomposes the value function $\nu^{tot}(\textbf{s})$ (or global Lagrange multipliers) into local values using mixing network architectures. Let $\boldsymbol{\nu}(\textbf{s})=\{\nu_{1}(o_{1}),\ldots,\nu_{n}(o_{n})\}$ represent a collection of local “value functions” and let $\textbf{A}_{\boldsymbol{\nu}}(\textbf{s},\textbf{a})=\{A_{i}(o_{i},a_{i}),~{}i
=1,...,n\}$ represent a collection of local advantage functions. The local advantage functions are computed as $A_{i}(o_{i},a_{i})=q_{i}(o_{i},a_{i})-\nu_{i}(o_{i})$ for all $i\in{\mathcal{N}}$, where $\textbf{q}(\textbf{s},\textbf{a})=\{q_{i}(o_{i},a_{i}),~{}i=1,...,n\}$ is a vector of local Q-functions. To facilitate centralized learning, we create a mixing network, ${\mathcal{M}}_{\theta}$, where $\theta$ are the learnable weights, that aggregates the local values to form the global value and advantage functions as follows:

$$ $\nu^{tot}(\textbf{s},\textbf{a})={\mathcal{M}}_{\theta}[\boldsymbol{\nu}( \textbf{s})],\quad A^{tot}_{\nu}(\textbf{s},\textbf{a})={\mathcal{M}}_{\theta} [\textbf{q}(\textbf{s},\textbf{a})-\boldsymbol{\nu}(\textbf{s})],$ $$

where each network takes the vectors $\boldsymbol{\nu}(\textbf{s})$ or $\textbf{A}_{\boldsymbol{\nu}}(\textbf{s},\textbf{a})$ as inputs and outputs $\nu^{tot}$ and $A^{tot}_{\nu}$, respectively. Under this architecture, the learning objective becomes:

$$ $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)=(1-\gamma)\mathbb{E}_{ \textbf{s}\sim p_{0}}[{\mathcal{M}}_{\theta}[\boldsymbol{\nu}(\textbf{s})]]+ \mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[\alpha f^{*} \left(\frac{{\mathcal{M}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a})- \boldsymbol{\nu}(\textbf{s})]}{\alpha}\right)\right],$ $$

with the observation that $\textbf{A}_{\nu}(\textbf{s},\textbf{a})$ can be expressed as a linear function of $\boldsymbol{\nu}$. There are different ways to construct the mixing network ${\mathcal{M}}_{\theta}$; prior works often employ a single linear combination (1-layer network) or a two-layer network with convex activations such as ReLU, ELU, or Maxout. In the following, we show a general result stating that the learning objective function is convex in $\boldsymbol{\nu}$, provided that the mixing network is constructed with non-negative weights and convex activations.

###### Theorem 4.2 .

If the mixing network ${\mathcal{M}}_{\theta}[\cdot]$ is constructed with non-negative weights and convex activations, then $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$ is convex in $\boldsymbol{\nu}$.

Theorem 4.2 shows that $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$ is convex in $\boldsymbol{\nu}$ when using any multi-layer feed-forward mixing networks with non-negative weights and convex activation functions. This finding is highly general and non-trivial, given the nonlinearity and complexity of both the function (in terms of $\boldsymbol{\nu}$) and the mixing networks. Previous work has often focused on single-layer (Wang et al., ) or two-layer mixing structures (Rashid et al., ; Bui et al., ), emphasizing that such two-layer networks can approximate any monotonic function arbitrarily closely as network width approaches infinity (Dugas et al., ). In our experiments, we test two configurations for the mixing network: a linear combination (or 1-layer) and a 2-layer feed-forward network. While 2-layer mixing structures have shown strong performance in online MARL (Rashid et al., ; Son et al., ; Wang et al., ), we observe in our offline settings that the linear combination approach provides more stable results.

###### Theorem 4.2 .

### 4.3 Policy Extraction

Let $\boldsymbol{\nu}^{*}$ be an optimal solution to the training problem with mixing networks, i.e.,

$$ $\min\limits_{\boldsymbol{\nu},\theta}\widetilde{{\mathcal{L}}}(\boldsymbol{\nu },\theta).$ (7) $$

We now need to extract a local and joint policy from this solution.
Based on Prop. 4.1, given $\boldsymbol{\nu}^{*}$, we can compute this occupancy ratio as follows: :

$$ $w^{tot*}(\textbf{s},\textbf{a})=\max\left\{0,{f^{\prime}}^{-1}\left(\frac{{ \mathcal{M}}_{\theta}[\textbf{A}_{\boldsymbol{\nu}^{*}}(\textbf{s},\textbf{a}) ]}{\alpha}\right)\right\}.$ $$

The global policy can then be obtained as follows:
$\boldsymbol{\pi}^{*}_{tot}(\textbf{a}|\textbf{s})=\frac{w^{tot*}(\textbf{s},
\textbf{a})\cdot\rho^{\mu_{tot}}(\textbf{s},\textbf{a})}{\sum_{\textbf{a}^{
\prime}\in\mathcal{A}}w^{tot*}(\textbf{s},\textbf{a}^{\prime})\cdot\rho^{\mu_{
tot}}(\textbf{s},\textbf{a}^{\prime})}.$
This computation, however, is not practical since $\rho^{\mu_{tot}}$ is generally not available and might not be accurately estimated in the offline setting. A more practical way to estimate the global policy, $\boldsymbol{\pi}^{*}_{tot}$, as the result of solving the following weighted behavioral cloning (BC):

$$ $\displaystyle\max_{\boldsymbol{\pi}_{tot}\in\Pi_{tot}}\mathbb{E}_{(\textbf{s}, \textbf{a})\sim\rho^{\boldsymbol{\pi}^{*}_{tot}}}[\log\boldsymbol{\pi}_{tot}( \textbf{a}|\textbf{s})]=\max_{\boldsymbol{\pi}_{tot}\in\Pi_{tot}}\mathbb{E}_{( \textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}[w^{tot*}(\textbf{s},\textbf{a}) \log\boldsymbol{\pi}_{tot}(\textbf{a}|\textbf{s})]$ (8) $$

where $\Pi_{tot}$ represents the feasible set of global policies. Here we assume that $\Pi_{tot}$ contains decomposable global policies, i.e., $\Pi_{tot}=\{\boldsymbol{\pi}_{tot}\mid\exists\pi_{i},~{}\forall i\in\mathcal{N
}\text{ such that }\boldsymbol{\pi}_{tot}(\textbf{a}|\textbf{s})=\prod_{i\in
\mathcal{N}}\pi_{i}(a_{i}|o_{i})\}$. In other words, $\Pi_{tot}$ consists of global policies that can be expressed as a product of local policies. This decomposability is highly useful for decentralized learning and has been widely adopted in MARL (Wang et al., ; Bui et al., ; Zhang et al., ).

While the above weighted BC appears practical, as $(\textbf{s},\textbf{a})$ can be sampled from the offline dataset generated by $\rho^{\boldsymbol{\pi}_{tot}}$, and since $w^{tot*}(\textbf{s},\textbf{a})$ is available from solving 7, it does not directly yield local policies, which are essential for decentralized execution. To address this, we propose solving the following weighted BC for each local agent $i\in\mathcal{N}$:

$$ $\max\nolimits_{\pi_{i}}\mathbb{E}_{(\textbf{s},\textbf{a})\sim\mathcal{D}} \left[w^{tot*}(\textbf{s},\textbf{a})\log\pi_{i}(a_{i}|o_{i})\right].$ $$

This local WBC approach has several attractive properties. First, $w^{tot*}(\textbf{s},\textbf{a})$ explicitly appears in the local policy optimization and is computed from global observations and actions. This enables local policies to be optimized with global information, ensuring consistency with the credit assignment in the multi-agent system. Furthermore, as shown in Proposition 4.3 below, the optimization of local policies through the lobcal WBC is highly consistent with the global weighted BC in 8.

###### Proposition 4.3 .

Let $\pi^{*}$ be the optimal solution to 4.3. Then $\bm{\pi}^{*}_{tot}(\textbf{a}|\textbf{s})=\prod_{i\in\mathcal{N}}\pi^{*}_{i}(a
_{i}|o_{i})$ is also optimal for the global weighted BC in 8.

Here we note that consistency between global and local policies is a critical aspect of centralized training with CTDE. Previous MARL approaches typically achieve this by factorizing the Q- or V-functions into local functions, and training local policies based on these local ones (Rashid et al., ; Wang et al., ; Bui et al., ). However, in our case, there are key differences that prevent us from employing such local values to derive local policies. Specifically, we factorize the Lagrange multipliers $\nu^{tot}$ to train the stationary distribution ratio $w^{tot}$. While local $w$ values can be extracted from the local $\nu_{i}$, these local $w$ values do not represent a local stationary distribution ratio and therefore cannot be used to recover local policies.

###### Proposition 4.3 .

## 5 Practical Algorithm

Figure: Algorithm 1 ComaDICE: Offline Cooperative MARL with Stationary DIstribution Correction Estimation

Let ${\mathcal{D}}$ represent the offline dataset, consisting of sequences of local observations and actions gathered from a global behavior policy $\boldsymbol{\pi}_{tot}$. To train the value function $\boldsymbol{\nu}$, we construct a value network $\nu_{i}(o_{i};\psi_{\nu})$ for each local agent $i$, along with a network for each local Q-function $q_{i}(o_{i},a_{i};\psi_{q})$, where $\psi_{\nu}$ and $\psi_{q}$ are learnable parameters for the local value and Q-functions. Each local advantage function is then calculated as follows:
The global value function and advantage function are subsequently aggregated using two mixing networks with a shared set of learnable parameters $\theta$:

$$ $\nu^{tot}(\textbf{s})={\mathcal{M}}^{\textbf{s}}_{\theta}[\boldsymbol{\nu}( \textbf{s};\psi_{\nu})],\quad A^{tot}_{\nu}(\textbf{s},\textbf{a})={\mathcal{M }}^{\textbf{s}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a};\psi_{q})- \boldsymbol{\nu}(\textbf{s};\psi_{\nu})],$ $$

where ${\mathcal{M}}^{\textbf{s}}_{\theta}[\cdot]$ represents a linear combination of its inputs with non-negative weights, such that ${\mathcal{M}}^{\textbf{s}}_{\theta}[\boldsymbol{\nu}(\textbf{s};\psi_{\nu})]=
\boldsymbol{\nu}(\textbf{s};\psi_{\nu})^{\top}W^{\textbf{s}}_{\theta}+b^{
\textbf{s}}_{\theta}$, where $W^{\textbf{s}}_{\theta}$ and $b^{\textbf{s}}_{\theta}$ are weights of the mixing network.(^1^11In our experiments, we use a single-layer mixing network due to its superior performance compared to a two-layer structure, though our approach is general and can handle any multi-layer feed-forward mixing network.) It is important to note that $W^{\textbf{s}}_{\theta}$ and $b^{\textbf{s}}_{\theta}$ are generated by hyper-networks that take the global state s and the learnable parameters $\theta$ as inputs. In this context, we employ the same mixing network ${\mathcal{M}}^{\textbf{s}}_{\theta}$ to combine the local values and advantages. However, our framework is flexible enough to allow the use of two different mixing networks for $\nu^{tot}$ and $A_{\nu}^{tot}$.

In our setting, the relationship between the global Q-function, value, and advantage functions is described in Eq. 5. Specifically, we have:
$A^{tot}_{\nu}(\textbf{s},\textbf{a})=r({\mathcal{Z}}(\textbf{s}),\textbf{a})+
\gamma\mathbb{E}_{\textbf{s}^{\prime}\sim P(\cdot|\textbf{s},\textbf{a})}[\nu^
{tot}(\textbf{s}^{\prime})]-\nu^{tot}(\textbf{s}).$
To capture this relationship, we train the Q-function by optimizing the following MSE loss:

$$ $\min\nolimits_{\textbf{q}}\sum\nolimits_{(\textbf{s},\textbf{a},\textbf{s}^{ \prime})\sim{\mathcal{D}}}\left(A^{tot}_{\nu}(\textbf{s},\textbf{a})-r({ \mathcal{Z}}(\textbf{s}),\textbf{a})+\gamma\nu^{tot}(\textbf{s}^{\prime})-\nu^ {tot}(\textbf{s})\right)^{2}.$ $$

This is equivalent to:

$$ $\displaystyle\min\nolimits_{\psi_{q}}{\mathcal{L}}_{q}(\psi_{q})=\sum\nolimits _{(\textbf{s},\textbf{a},\textbf{s}^{\prime})\sim{\mathcal{D}}}$ $\displaystyle\Big{(}{\mathcal{M}}^{\textbf{s}}_{\theta}[\textbf{q}(\textbf{s}, \textbf{a};\psi_{q})-\boldsymbol{\nu}(\textbf{s};\psi_{\nu})]$ $\displaystyle-r({\mathcal{Z}}(\textbf{s}),\textbf{a})+\gamma{\mathcal{M}}^{ \textbf{s}^{\prime}}_{\theta}[\boldsymbol{\nu}(\textbf{s}^{\prime};\psi_{\nu}) ]-{\mathcal{M}}^{\textbf{s}}_{\theta}[\boldsymbol{\nu}(\textbf{s};\psi_{\nu})] \Big{)}^{2}$ (9) $$

For the primary loss function used to train the value function, we leverage transitions from the offline dataset to approximate the objective $\widetilde{{\mathcal{L}}}$, resulting in the following loss function for offline training:

$$ $\widetilde{{\mathcal{L}}}(\psi_{\nu},\theta)=(1-\gamma)\mathbb{E}_{\textbf{s}_ {0}\sim{\mathcal{D}}}[{\mathcal{M}}^{\textbf{s}_{0}}_{\theta}[\boldsymbol{\nu} (\textbf{s}_{0};\psi_{\nu})]]+\mathbb{E}_{(\textbf{s},\textbf{a})\sim{\mathcal {D}}}\left[\alpha f^{*}\!\!\left(\frac{{\mathcal{M}}^{\textbf{s}}_{\theta}[ \textbf{q}(\textbf{s},\textbf{a};\psi_{q})-\boldsymbol{\nu}(\textbf{s};\psi_{ \nu})]}{\alpha}\right)\right]$ (10) $$

As mentioned, after obtaining $(\boldsymbol{\nu}^{*},\theta^{*})$ by solving $\min_{\psi_{\nu},\theta}\widetilde{{\mathcal{L}}}(\psi_{\nu},\theta)$, we compute the occupancy ratio:
$w^{tot*}_{\nu}(\textbf{s},\textbf{a})=\max\left\{0,{f^{\prime}}^{-1}\left(
\frac{{\mathcal{M}}^{\textbf{s}}_{\theta^{*}}[\boldsymbol{\nu}^{*}(\textbf{s})
]-{\mathcal{M}}^{\textbf{s}}_{\theta^{*}}[\textbf{q}(\textbf{s},\textbf{a};
\psi_{q})]}{\alpha}\right)\right\}.$
To train the local policy $\pi_{i}(a_{i}|o_{i})$, we represent it using a policy network $\pi_{i}(a_{i}|o_{i};\eta_{i})$, where $\eta_{i}$ are the learnable parameters. The training process involves optimizing the following weighted behavioral cloning (BC) objective:

$$ $\max\nolimits_{\eta_{i}}~{}~{}{\mathcal{L}}_{\pi}(\eta_{i})=\sum\nolimits_{( \textbf{s},\textbf{a})\sim{\mathcal{D}}}w^{tot*}_{\nu}(\textbf{s},\textbf{a}) \log(\pi_{i}(a_{i}|o_{i};\eta_{i})).$ (11) $$

Our ComaDICE algorithm consists of two primary steps. The first step involves estimating the occupancy ratio $w^{tot*}$ from the offline dataset. The second step focuses on training the local policy by solving the weighted BC problem using $w^{tot*}$. In the first step, we simultaneously update the Q-functions $\psi_{q}$, the mixing network parameters $\theta$, and the value function $\psi_{\nu}$, aiming to minimize the mean squared error (MSE) in Eq. 9 while optimizing the main loss function in Eq. 10.

## 6 Experiments

### 6.1 Environments

We utilize three standard MARL environments: SMACv1 (Samvelyan et al., ), SMACv2 (Ellis et al., ), and Multi-Agent MuJoCo (MaMujoco) (de Witt et al., ), each offering unique challenges and configurations for evaluating cooperative MARL algorithms.

SMACv1. SMACv1 is based on Blizzard’s StarCraft II. It uses the StarCraft II API and DeepMind’s PySC2 to enable agent interactions with the game. SMACv1 focuses on decentralized micromanagement scenarios where each unit is controlled by an RL agent. Tasks like *2c_vs_64zg* and *5m_vs_6m* are labeled hard, while *6h_vs_8z* and *corridor* are super hard. The offline dataset, provided by Meng et al. (), was generated using MAPPO-trained agents (Yu et al., ).

SMACv2. In comparison to SMACv1, SMACv2 introduces increased randomness and diversity by randomizing start positions, unit types, and modifying sight and attack ranges. This version includes tasks such as *protoss*, *terran*, and *zerg*, with instances ranging from *5_vs_5* to *20_vs_23*, increasing in difficulty. Our offline dataset for SMACv2 was generated by running MAPPO for 10 million training steps and collecting 1,000 trajectories, ensuring medium quality but comprehensive coverage of the learning process. To the best of our knowledge, we are the first to explore SMACv2 in offline MARL, whereas most prior work has used this environment in online settings.

MaMujoco. MaMujoco serves as a benchmark for continuous cooperative multi-agent robotic control. Derived from the single-agent MuJoCo control suite in OpenAI Gym (Brockman et al., ), it presents scenarios where multiple agents within a single robot must collaborate to achieve tasks. The tasks include *Hopper-v2*, *Ant-v2*, and *HalfCheetah-v2*, with instances labeled as *expert*, *medium*, *medium-replay*, and *medium-expert*. The offline dataset was created by (Wang et al., ) using the HAPPO method (Wang et al., ).

### 6.2 Baselines

We consider the following baselines, which represent either standard or state-of-the-art (SOTA) methods for offline MARL: (i) BC (Behavioral Cloning); (ii) BCQ (Batch-Constrained Q-learning) (Fujimoto et al., ) – an offline RL algorithm that constrains the policy to actions similar to those in the dataset to reduce distributional shift, adapted for offline MARL settings; (iii) CQL (Conservative Q-Learning) (Kumar et al., ) – a method that stabilizes offline Q-learning by penalizing out-of-distribution actions, ensuring conservative value estimates; (iv) ICQ (Implicit Constraint Q-learning) (Yang et al., ) – an approach using importance sampling to manage out-of-distribution actions in multi-agent settings; (v) OMAR (Offline MARL with Actor Rectification) (Pan et al., ) – a method combining CQL with optimization techniques to ensure the global validity of local regularizations, promoting cooperative behavior; (vi) OMIGA (Offline MARL with Implicit Global-to-Local Value Regularization) (Wang et al., ) – a SOTA method that transforms global regularizations into implicit local ones, optimizing local policies with global insights.

**Table 1: Comparison of winrates for ComaDICE and baselines across SMACv2 tasks.**
| Instances | BC | BCQ | CQL | ICQ | OMAR | OMIGA | ComaDICE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (ours) |  |  |  |  |  |  |  |  |
| Protoss | 5_vs_5 | 36.9±8.7 | 16.2±2.3 | 10.0±4.1 | 36.9±9.1 | 21.2±4.1 | 33.1±5.4 | 46.2±6.1 |
| 10_vs_10 | 36.2±10.6 | 9.4±5.6 | 26.2±7.6 | 28.1±6.6 | 13.8±7.0 | 40.0±10.7 | 50.6±8.7 |  |
| 10_vs_11 | 19.4±4.6 | 10.0±4.1 | 10.6±5.4 | 12.5±4.4 | 12.5±3.4 | 16.2±6.1 | 20.0±4.2 |  |
| 20_vs_20 | 37.5±4.4 | 6.2±2.0 | 11.9±4.1 | 32.5±8.1 | 23.8±2.5 | 36.2±5.1 | 47.5±7.8 |  |
| 20_vs_23 | 13.8±1.5 | 1.2±1.5 | 0.0±0.0 | 12.5±5.6 | 11.2±7.8 | 12.5±8.1 | 13.8±5.8 |  |
| Terran | 5_vs_5 | 30.0±4.2 | 12.5±6.2 | 9.4±7.9 | 23.1±5.8 | 14.4±4.7 | 28.1±4.4 | 30.6±8.2 |
| 10_vs_10 | 29.4±5.8 | 6.9±6.1 | 9.4±5.6 | 16.9±5.8 | 15.0±4.6 | 29.4±3.2 | 32.5±5.8 |  |
| 10_vs_11 | 16.2±3.6 | 3.8±4.6 | 7.5±6.4 | 5.0±4.2 | 9.4±5.6 | 12.5±5.2 | 19.4±5.4 |  |
| 20_vs_20 | 26.2±10.4 | 5.0±3.2 | 10.6±4.2 | 15.6±3.4 | 7.5±7.3 | 21.9±4.4 | 29.4±3.8 |  |
| 20_vs_23 | 4.4±4.2 | 0.0±0.0 | 0.0±0.0 | 7.5±6.1 | 5.0±4.2 | 4.4±2.5 | 9.4±5.2 |  |
| Zerg | 5_vs_5 | 26.9±10.0 | 14.4±4.2 | 14.4±5.8 | 18.8±7.1 | 13.8±6.1 | 21.9±5.9 | 31.2±7.7 |
| 10_vs_10 | 25.0±2.8 | 5.6±4.6 | 5.6±4.6 | 15.6±7.4 | 19.4±2.3 | 23.8±6.4 | 33.8±11.8 |  |
| 10_vs_11 | 13.8±4.7 | 9.4±5.2 | 6.2±4.4 | 10.6±6.7 | 10.6±3.8 | 13.8±6.7 | 19.4±3.6 |  |
| 20_vs_20 | 8.1±1.5 | 2.5±1.2 | 1.2±1.5 | 10.0±7.8 | 12.5±4.4 | 10.0±2.3 | 9.4±6.2 |  |
| 20_vs_23 | 7.5±3.2 | 0.6±1.3 | 1.2±1.5 | 7.5±3.2 | 3.8±2.3 | 4.4±4.2 | 11.2±4.2 |  |

We used experimental results contributed by the authors of OMIGA (Wang et al., ) as our baselines. They provided both the results and source code for all the baseline methods. This source code was also employed to run these baselines for the SMACv2 environment. All hyperparameters were kept at their default settings, and each experiment was conducted with five different random seeds to ensure robustness and reproducibility of the results.

### 6.3 Main Comparison

We now present a comprehensive evaluation of our proposed algorithm, ComaDICE, against several baseline methods in offline MARL. The baselines selected for comparison include both standard and SOTA approaches, providing a robust benchmark to assess the effectiveness of ComaDICE.

Our evaluation focuses on two primary metrics: returns and winrates. Returns are the average rewards accumulated by the agents across multiple trials, providing a measure of policy effectiveness. Winrates, applicable in competitive environments such as SMACv1 and SMACv2, indicate the success rate of agents against opponents, reflecting the algorithm’s robustness in adversarial settings.

The experimental results, summarized in Tables 1 and 2, demonstrate that ComaDICE consistently achieves superior performance compared to baseline methods across a range of scenarios. Notably, ComaDICE excels in complex tasks, highlighting its ability to effectively manage distributional shifts in challenging environments.

Figures 1 illustrates the learning curves for each algorithm, showing that ComaDICE not only outperforms other methods in terms of mean returns but also exhibits lower variance, indicating more stable and reliable performance. These findings underscore the robustness and adaptability of ComaDICE, setting a new benchmark for offline MARL.

**Table 2: Average returns for ComaDICE and baselines on MaMuJoCo benchmarks.**
| Instances | BCQ | CQL | ICQ | OMIGA | ComaDICE |  |
| --- | --- | --- | --- | --- | --- | --- |
| (ours) |  |  |  |  |  |  |
| Hopper | expert | 77.9±58.0 | 159.1±313.8 | 754.7±806.3 | 859.6±709.5 | 2827.7±62.9 |
| medium | 44.6±20.6 | 401.3±199.9 | 501.8±14.0 | 1189.3±544.3 | 822.6±66.2 |  |
| m-replay | 26.5±24.0 | 31.4±15.2 | 195.4±103.6 | 774.2±494.3 | 906.3±242.1 |  |
| m-expert | 54.3±23.7 | 64.8±123.3 | 355.4±373.9 | 709.0±595.7 | 1362.4±522.9 |  |
| Ant | expert | 1317.7±286.3 | 1042.4±2021.6 | 2050.0±11.9 | 2055.5±1.6 | 2056.9±5.9 |
| medium | 1059.6±91.2 | 533.9±1766.4 | 1412.4±10.9 | 1418.4±5.4 | 1425.0±2.9 |  |
| m-replay | 950.8±48.8 | 234.6±1618.3 | 1016.7±53.5 | 1105.1±88.9 | 1122.9±61.0 |  |
| m-expert | 1020.9±242.7 | 800.2±1621.5 | 1590.2±85.6 | 1720.3±110.6 | 1813.9±68.4 |  |
| Half<br>Cheetah | expert | 2992.7±629.7 | 1189.5±1034.5 | 2955.9±459.2 | 3383.6±552.7 | 4082.9±45.7 |
| medium | 2590.5±1110.4 | 1011.3±1016.9 | 2549.3±96.3 | 3608.1±237.4 | 2664.7±54.2 |  |
| m-replay | -333.6±152.1 | 1998.7±693.9 | 1922.4±612.9 | 2504.7±83.5 | 2855.0±242.2 |  |
| m-expert | 3543.7±780.9 | 1194.2±1081.0 | 2834.0±420.3 | 2948.5±518.9 | 3889.7±81.6 |  |

Figure: Figure 1: Evaluation curves of ComaDICE and baselines over time on SMACv2 tasks.
Refer to caption: x1.png

### 6.4 Ablation Study - Impact of the Regularization Parameter Alpha

We investigate how varying the regularization parameter alpha ($\alpha$) affects the performance of our ComaDICE algorithm. The parameter $\alpha$ is crucial for balancing the trade-off between maximizing rewards and penalizing deviations from the offline dataset’s distribution. We conducted experiments with $\alpha$ values ranging from $\{0.01,0.1,1,10,100\}$, evaluating performance using winrates in the SMACv2 environment and returns in the MaMujoco environment. These results, illustrated in Figure 2, highlight the sensitivity of ComaDICE to different $\alpha$ values. In particular, we observe that ComaDICE achieves optimal performance when $\alpha$ is around 10, suggesting that the stationary distribution regularizer plays a essential role in the success of our algorithm.

In our appendix, we provide additional ablation studies to analyze the performance of our algorithm using different forms of f-divergence functions, as well as comparisons between 1-layer and 2-layer mixing network structures. The appendix also includes proofs of the theoretical claims made in the main paper, details of our experimental settings, and other experimental information.

Figure: Figure 2: Impact of regularization parameter $\alpha$ on performance in different environments.
Refer to caption: x17.png

## 7 Conclusion, Future Work and Broader Impacts

Conclusion. In this paper, we propose ComaDICE, a principled framework for offline MARL. Our algorithm incorporates a stationary distribution shift regularizer into the standard MARL objective to address the conventional distribution shift issue in offline RL. To facilitate training within a CTDE framework, we decompose both the global value and advantage functions using a mixing network. We demonstrate that, under our mixing architecture, the main objective function is concave in the value function, which is crucial for ensuring stable and efficient training. The results of this training are then utilized to derive local policies through a weighted BC approach, ensuring consistency between global and local policy optimization. Extensive experiments on SOTA benchmark tasks, including SMACv2, show that ComaDICE outperforms other baseline methods.

Limitations and Future Work: There are some limitations that are not addressed within the scope of this paper. For instance, we focus solely on cooperative learning, leaving open the question of how the approach would perform in cooperative-competitive settings. Extending ComaDICE to such scenarios would require considerable effort and is an interesting direction for future research. Additionally, in our training objective, the DICE term is designed to reduce the divergence between the learning policy and the behavior policy. As a result, the performance of the algorithm is heavily dependent on the quality of the behavior policy. Although this reliance may be unavoidable, future research should focus on mitigating the influence of the behavior policy on training outcomes. Furthermore, our algorithm, like other baselines, still requires a large amount of data to achieve desirable learning outcomes. Improving sample efficiency would be another valuable area for future research.

Broader Impacts: The development of an offline MARL algorithm using a stationary distribution shift regularizer could lead to improved performance in tasks where real-time interaction is costly, such as robotics, autonomous driving, and healthcare. It could also promote safer exploration and wider adoption of offline learning in high-stakes environments. On the negative side, since the algorithm relies heavily on the behavior policy, if the behavior policy is flawed or biased, the performance of the learnt policy could also suffer. This could reinforce preexisting biases or suboptimal behaviors in real-world applications. Moreover, like any AI technology, there is a risk of the algorithm being applied in unintended or harmful ways, such as in surveillance or military applications, where multi-agent systems could be used to manipulate environments or people without adequate oversight.

## Ethical Statement

Our work introduces ComaDICE, a framework for offline MARL, aimed at improving training stability and policy optimization in complex multi-agent environments. While this research has significant potential for positive applications, particularly in domains such as autonomous systems, resource management, and multi-agent simulations, it is crucial to address the ethical implications and risks associated with this technology.

The deployment of reinforcement learning systems in real-world, multi-agent settings raises concerns about unintended behaviors, especially in safety-critical domains. If the policies learned by ComaDICE are applied without proper testing and validation, they may lead to undesirable or harmful outcomes, especially in areas such as autonomous driving, healthcare, or robotics. Additionally, bias in the training data or simulation environments could result in suboptimal policies that unfairly impact certain agents or populations, potentially leading to ethical concerns regarding fairness and transparency.

To mitigate these risks, we emphasize the need for extensive testing and validation of policies generated using ComaDICE, particularly in real-world environments where the consequences of errors could be severe. It is also essential to ensure that the datasets and simulations used in training are representative, unbiased, and carefully curated. We encourage practitioners to use human oversight and collaborate with domain experts to ensure that ComaDICE is applied responsibly, particularly in high-stakes settings.

## Reproducibility Statement

In order to facilitate reproducibility, we have submitted the source code for ComaDICE, along with the datasets utilized to produce the experimental results presented in this paper (all these will be made publicly available if the paper gets accepted).
Additionally, in the appendix, we provide details of our algorithm, including key implementation steps and details needed to replicate the results. The hyper-parameter settings for all experiments are also included to ensure that others can reproduce the findings under the same experimental conditions. We invite the research community to explore and apply the ComaDICE framework in various environments to further validate and expand upon the results reported in this work.

## Appendix A Missing Proofs

### A.1 Proof of Proposition 4.1

Proposition. The minimax problem in 6 is equivalent to
$\min_{\nu^{tot}}~{}~{}\left\{\widetilde{{\mathcal{L}}}(\nu^{tot})\right\}$, where

$$ $\widetilde{{\mathcal{L}}}(\nu^{tot})=(1-\gamma)\mathbb{E}_{\textbf{s}\sim p_{0 }}[\nu^{tot}(\textbf{s})]+\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{ tot}}}\left[\alpha f^{*}\left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{ \alpha}\right)\right]$ $$

where $f^{*}$ is convex conjugate of $f$, i.e., $f^{*}(y)=\sup_{t\geq 0}\{ty-f(t)\}$. Moreover, if $\nu^{tot}$ is parameterized by $\theta$, the first order derivative of $\widetilde{{\mathcal{L}}}(\nu^{tot})$ w.r.t. $\theta$ is given as

$$ $\nabla_{\theta}\widetilde{{\mathcal{L}}}(\nu^{tot})=(1-\gamma)\mathbb{E}_{ \textbf{s}\sim p_{0}}[\nabla_{\theta}\nu^{tot}(\textbf{s})]+\mathbb{E}_{( \textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[\nabla_{\theta}A^{tot}_{\nu}( \textbf{s},\textbf{a})w^{tot*}_{\nu}(\textbf{s},\textbf{a})\right]$ $$

where $w^{tot*}_{\nu}(s,a)=\max\{0,{f^{\prime}}^{-1}(A^{tot}_{\nu}(\textbf{s},\textbf
{a})/\alpha)\}$, where $f^{\prime-1}(\cdot)$ is the inverse function of the first-order derivative of $f$.

###### Proof.

We write the Lagrange dual function as:

$$ $\displaystyle{\mathcal{L}}(\nu^{tot},$ $\displaystyle\rho^{\boldsymbol{\pi}_{tot}})=\mathbb{E}_{(\textbf{o},\textbf{a} )\sim\rho^{\boldsymbol{\pi}_{tot}}}\left[r(\textbf{o},\textbf{a})\right]- \alpha\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[f\left( \frac{\rho^{\boldsymbol{\pi}_{tot}}(\textbf{s},\textbf{a})}{\rho^{\mu_{tot}}( \textbf{s},\textbf{a})}\right)\right]$ $\displaystyle-\sum_{\textbf{s}}\nu^{tot}(\textbf{s})\left(\sum_{\textbf{a}^{ \prime}}\rho^{\boldsymbol{\pi}_{tot}}(\textbf{s},\textbf{a}^{\prime})-(1- \gamma)p_{0}(\textbf{s})-\gamma\sum_{\textbf{a}^{\prime},\textbf{s}^{\prime}} \rho^{\boldsymbol{\pi}_{tot}}(\textbf{s}^{\prime},\textbf{a}^{\prime})P( \textbf{s}|\textbf{a}^{\prime},\textbf{s}^{\prime})\right)$ $\displaystyle=\sum_{\textbf{s}}\nu^{tot}(\textbf{s})(1-\gamma)p_{0}(\textbf{s} )-\alpha\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[f\left( \frac{\rho^{\boldsymbol{\pi}_{tot}}(\textbf{s},\textbf{a})}{\rho^{\mu_{tot}}( \textbf{s},\textbf{a})}\right)\right]$ $\displaystyle~{}~{}~{}~{}~{}~{}+\sum_{\textbf{s},\textbf{a}}\rho^{\mu_{tot}}( \textbf{s},\textbf{a})\left(r(\textbf{o},\textbf{a})+\gamma\mathbb{E}_{\textbf {s}^{\prime}\sim P(\cdot|\textbf{s},\textbf{a})}\nu^{tot}(\textbf{s}^{\prime}) -\nu^{tot}(\textbf{s})\right)$ $\displaystyle=(1-\gamma)\mathbb{E}_{\textbf{s}\sim p_{0}}[\nu^{tot}(\textbf{s} )]+\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[-\alpha f \left(w^{tot}_{\nu}(\textbf{s},\textbf{a})\right)+w^{tot}_{\nu}(\textbf{s}, \textbf{a})A^{tot}_{\nu}(\textbf{s},\textbf{a})\right]$ (12) $$

where $w^{tot}_{\nu}(\textbf{s},\textbf{a})=\frac{\rho^{\boldsymbol{\pi}_{tot}}(
\textbf{s},\textbf{a})}{\rho^{\mu_{tot}}(\textbf{s},\textbf{a})}$. We now see that, for each $(\textbf{s},\textbf{a})$, each component $-\alpha f\left(w^{tot}_{\nu}(\textbf{s},\textbf{a})\right)+w^{tot}_{\nu}(
\textbf{s},\textbf{a})A^{tot}_{\nu}(\textbf{s},\textbf{a})$ is maximized at:

$$ $\max_{w^{tot}\geq 0}-\alpha f\left(w^{tot}_{\nu}(\textbf{s},\textbf{a})\right) +w^{tot}_{\nu}(\textbf{s},\textbf{a})A^{tot}_{\nu}(\textbf{s},\textbf{a})=f^{* }\left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha}\right)$ $$

where $f^{*}$ is the (variant) convex conjugate of the convex function $f$. We then obtain:

$$ $\max_{w^{tot}\geq 0}{\mathcal{L}}(\nu^{tot},w^{tot})=\widetilde{{\mathcal{L}}} (\nu^{tot})=(1-\gamma)\mathbb{E}_{\textbf{s}\sim p_{0}}[\nu^{tot}(\textbf{s})] +\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[\alpha f^{*} \left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha}\right)\right]$ $$

Moreover, consider the maximization problem $\max_{w^{tot}\geq 0}T(w^{tot}(\textbf{s},\textbf{a}))=-\alpha f\left(w^{tot}_{
\nu}(\textbf{s},\textbf{a})\right)+w^{tot}_{\nu}(\textbf{s},\textbf{a})A^{tot}
_{\nu}(\textbf{s},\textbf{a})$. Taking its first-order derivative w.r.t $w^{tot}(\textbf{s},\textbf{a})$ yields:

$$ $-\alpha f^{\prime}(w^{tot}(\textbf{s},\textbf{a}))+A^{tot}_{\nu}(\textbf{s}, \textbf{a})$ $$

So, if $f^{\prime-1}\left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha}\right)\geq
0$, then $w^{tot*}(\textbf{s},\textbf{a})=f^{\prime-1}\left(\frac{A^{tot}_{\nu}(\textbf{
s},\textbf{a})}{\alpha}\right)\geq 0$ is optimal for the maximization problem. Otherwise, if $f^{\prime-1}\left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha}\right)<0$, we see that $T(w^{tot}(\textbf{s},\textbf{a}))$ is increasing when $w^{tot}(\textbf{s},\textbf{a})\leq f^{\prime-1}\left(\frac{A^{tot}_{\nu}(
\textbf{s},\textbf{a})}{\alpha}\right)$ and decreasing when $w^{tot}(\textbf{s},\textbf{a})\geq f^{\prime-1}\left(\frac{A^{tot}_{\nu}(
\textbf{s},\textbf{a})}{\alpha}\right)$, implying that the maximization problem has an optimal solution at $w^{tot*}(\textbf{s},\textbf{a})=0$. So, putting all together, $w^{tot*}_{\nu}(s,a)=\max\{0,{f^{\prime}}^{-1}(A^{tot}_{\nu}(\textbf{s},\textbf
{a})/\alpha)\}$ is optimal for the maximization problem $\max_{w^{tot}\geq 0}T(w^{tot}(\textbf{s},\textbf{a}))$.

To get derivatives of $\widetilde{{\mathcal{L}}}(\nu^{tot})$, we note that, for any $y\in\mathbb{R}$, $\nabla f^{*}(y)=t^{*}$, where $y^{*}=\text{argmax}_{t\geq 0}(ty-f(t))$. Thus, the first-order derivative of $f^{*}\left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha}\right)$ can be computed as:

$$ $\nabla_{\theta}f^{*}\left(\frac{A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha} \right)=\frac{\nabla_{\theta}A^{tot}_{\nu}(\textbf{s},\textbf{a})}{\alpha}w^{ tot^{*}}(\textbf{s},\textbf{a})$ $$

which implies:

$$ $\nabla_{\theta}\widetilde{{\mathcal{L}}}(\nu^{tot})=(1-\gamma)\mathbb{E}_{ \textbf{s}\sim p_{0}}[\nabla_{\theta}\nu^{tot}(\textbf{s})]+\mathbb{E}_{( \textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[\nabla_{\theta}A^{tot}_{\nu}( \textbf{s},\textbf{a})w^{tot*}_{\nu}(\textbf{s},\textbf{a})\right]$ $$

we complete the proof.
∎

###### Proof.

### A.2 Proof of Theorem 4.2

Theorem. Assume the mixing network ${\mathcal{M}}_{\theta}[\cdot]$ is constructed with non-negative weights and convex activations, then $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$ is convex in $\boldsymbol{\nu}$.

###### Proof.

We first introduce the following lemma, which is essential to validate the convexity of $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$.

###### Lemma A.1 .

If the mixing network are multi-level feed-forward, constructed with non-negative weights and convex activations, then ${\mathcal{M}}_{\theta}[\boldsymbol{\nu}(\textbf{s})]$ and ${\mathcal{M}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a})-\boldsymbol{\nu}(
\textbf{s})]$ are convex in $\boldsymbol{\nu}$

###### Proof.

To simplify the proof, we first prove a general result stating that if ${\mathcal{M}}_{\theta}[\textbf{X}]$ is a multi-level feed-forward network with non-negative weights and convex activations, then ${\mathcal{M}}_{\theta}[\textbf{X}]$ is convex in X. To start, we note that any $N$-layer feed-forward network with input X can be defined recursively as

$$ $\displaystyle F^{0}(\textbf{X})$ $\displaystyle=\textbf{X}$ (13) $\displaystyle F^{n}(\textbf{X})$ $\displaystyle=\sigma^{n}\Big{(}F^{n-1}(\textbf{X})\Big{)}\times W_{n}+b_{n},~{ }n=1,\ldots,N$ (14) $$

where $\sigma^{n}$ is a set of activation functions applied to each element of vector $F^{n-1}(\textbf{X})$, and $W_{n}$ and $b_{n}$ are the weights and biases, respectively, at layer $n$. Therefore, we will prove the result by induction, i.e., $F^{n}(\textbf{X})$ is convex and non-decreasing in X for $n=0,\ldots$. Here we note that $F^{n}(\textbf{X})$ is a vector, so when we say “$F^{n}(\textbf{X})$ is convex and non-decreasing in X,” it means each element of $F^{n}(\textbf{X})$ is convex and non-decreasing in X.

We first see that the claim indeed holds for $n=0$. Now let us assume that $F^{n-1}(\textbf{X})$ is convex and non-decreasing in X; we will prove that $F^{n}(\textbf{X})$ is also convex and non-decreasing in X. The non-decreasing property can be easily verified as we can see, given two vectors X and $\textbf{X}^{\prime}$ such that $\textbf{X}\geq\textbf{X}^{\prime}$ (element-wise comparison), we have the following chain of inequalities:

$$ $\displaystyle F^{n-1}(\textbf{X})$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\geq}}F^{n-1}(\textbf{X}^{\prime})$ $\displaystyle\sigma^{n}(F^{n-1}(\textbf{X}))$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\geq}}\sigma^{n}(F^{n-1}(\textbf{X} ^{\prime}))$ $\displaystyle\sigma^{n}(F^{n-1}(\textbf{X}))\times W_{n}+b_{n}$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{\geq}}\sigma^{n}(F^{n-1}(\textbf{X} ^{\prime}))\times W_{n}+b_{n}$ $$

where $(a)$ is due to the induction assumption that $F^{n-1}(\textbf{X})$ is non-decreasing in X, $(b)$ is because $\sigma^{n}$ is also non-decreasing, and $(c)$ is because the weights $W_{n}$ are non-negative.

To verify the convexity of $F^{n}(\textbf{X})$, we will show that for any $\textbf{X},\textbf{X}^{\prime}$, and any scalar $\alpha\in(0,1)$, the following holds:

$$ $\alpha F^{n}(\textbf{X})+(1-\alpha)F^{n}(\textbf{X})\geq F^{n}(\alpha\textbf{X }+(1-\alpha)\textbf{X}^{\prime})$ (15) $$

To this end, we write:

$$ $\displaystyle\alpha F^{n}(\textbf{X})+(1-\alpha)F^{n}(\textbf{X}^{\prime})$ $\displaystyle=\Big{(}\alpha\sigma^{n}(F^{n-1}(\textbf{X}))+(1-\alpha)\sigma^{n }(F^{n-1}(\textbf{X}^{\prime}))\Big{)}\times W_{n}+b_{n}$ $\displaystyle\stackrel{{\scriptstyle(d)}}{{\geq}}\Big{(}\sigma^{n}\Big{(} \alpha F^{n-1}(\textbf{X})+(1-\alpha)F^{n-1}(\textbf{X}^{\prime})\Big{)}\Big{) }\times W_{n}+b_{n}$ $\displaystyle\stackrel{{\scriptstyle(e)}}{{\geq}}\Big{(}\sigma^{n}\Big{(}F^{n- 1}(\alpha\textbf{X}+(1-\alpha)\textbf{X}^{\prime})\Big{)}\Big{)}\times W_{n}+b _{n}$ $\displaystyle=F^{n}(\alpha\textbf{X}+(1-\alpha)\textbf{X}^{\prime})$ $$

where $(d)$ is due to the assumption that activation functions $\sigma^{n}$ are convex and $W_{n}\geq 0$, and $(e)$ is because $\alpha F^{n-1}(\textbf{X})+(1-\alpha)F^{n-1}(\textbf{X}^{\prime})\geq F^{n-1}(
\alpha\textbf{X}+(1-\alpha)\textbf{X}^{\prime})$ (because $F^{n-1}(\textbf{X})$ is convex in X, by the induction assumption), and the activation functions $\sigma^{n}$ are non-decreasing and $W_{n}\geq 0$. So, we have:

$$ $\alpha F^{n}(\textbf{X})+(1-\alpha)F^{n}(\textbf{X}^{\prime})\geq F^{n}(\alpha \textbf{X}+(1-\alpha)\textbf{X}^{\prime})$ $$

implying that $F^{n}(\textbf{X})$ is convex in X. We then complete the induction proof and conclude that $F^{n}(\textbf{X})$ is convex and non-decreasing in X for any $n=0,\ldots,N$.

From the result above, since both $\boldsymbol{\nu}(\textbf{s})$ and $\textbf{q}(\textbf{s},\textbf{a})-\boldsymbol{\nu}(\textbf{s})$ are linear in $\boldsymbol{\nu}$, it follows that ${\mathcal{M}}_{\theta}[\boldsymbol{\nu}(\textbf{s})]$ and ${\mathcal{M}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a})-\boldsymbol{\nu}(
\textbf{s})]$ are convex with respect to $\boldsymbol{\nu}$.
∎

We are now ready to prove the convexity of $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$ with respect to $\boldsymbol{\nu}$. Directly verifying the convexity of this function is challenging, as it involves some complicated components such as $f^{*}\left(\frac{{\mathcal{M}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a})-
\boldsymbol{\nu}(\textbf{s})]}{\alpha}\right)$, which is difficult to analyze. However, we recall that:

$$ $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)=\max_{w^{tot}\geq 0}{ \mathcal{L}}(\boldsymbol{\nu},\theta,w^{tot}),$ $$

where

$$ $\displaystyle{\mathcal{L}}(\boldsymbol{\nu},\theta,w^{tot})$ $\displaystyle=(1-\gamma)\mathbb{E}_{\textbf{s}\sim p_{0}}[{\mathcal{M}}_{ \theta}[\boldsymbol{\nu}(\textbf{s})]]$ $\displaystyle\quad+\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}} \left[-\alpha f\left(w^{tot}_{\nu}(\textbf{s},\textbf{a})\right)+w^{tot}_{\nu} (\textbf{s},\textbf{a}){\mathcal{M}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a} )-\boldsymbol{\nu}(\textbf{s})]\right].$ $$

From Lemma A.1, we know that ${\mathcal{M}}_{\theta}[\boldsymbol{\nu}(\textbf{s})]$ and ${\mathcal{M}}_{\theta}[\textbf{q}(\textbf{s},\textbf{a})-\boldsymbol{\nu}(
\textbf{s})]$ are convex in $\boldsymbol{\nu}$, thus ${\mathcal{L}}(\boldsymbol{\nu},\theta,w^{tot})$ is also convex in $\boldsymbol{\nu}$. We now follow the standard approach to verify the convexity of $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$ as follows. Let $\boldsymbol{\nu}^{1}$ and $\boldsymbol{\nu}^{2}$ be two feasible value functions. Given any $\beta\in(0,1)$, we will prove that:

$$ $\beta\widetilde{{\mathcal{L}}}(\boldsymbol{\nu}^{1},\theta)+(1-\beta) \widetilde{{\mathcal{L}}}(\boldsymbol{\nu}^{2},\theta)\geq\widetilde{{\mathcal {L}}}(\beta\boldsymbol{\nu}^{1}+(1-\beta)\boldsymbol{\nu}^{2},\theta).$ (16) $$

To see why this should hold, we recall that ${\mathcal{L}}(\boldsymbol{\nu},\theta,w^{tot})$ is convex in $\boldsymbol{\nu}$ and $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)=\max_{w^{tot}\geq 0}{
\mathcal{L}}(\boldsymbol{\nu},\theta,w^{tot})$, leading to the following chain of inequalities:

$$ $\displaystyle\beta\widetilde{{\mathcal{L}}}(\boldsymbol{\nu}^{1},\theta)+(1- \beta)\widetilde{{\mathcal{L}}}(\boldsymbol{\nu}^{2},\theta)$ $\displaystyle=\beta\max_{w^{tot}}{\mathcal{L}}(\boldsymbol{\nu}^{1},\theta,w^{ tot})+(1-\beta)\max_{w^{tot}}{\mathcal{L}}(\boldsymbol{\nu}^{2},\theta,w^{tot})$ $\displaystyle\geq\max_{w^{tot}}\left\{\beta{\mathcal{L}}(\boldsymbol{\nu}^{1}, \theta,w^{tot})+(1-\beta){\mathcal{L}}(\boldsymbol{\nu}^{2},\theta,w^{tot})\right\}$ $\displaystyle\geq\max_{w^{tot}}\left\{{\mathcal{L}}(\beta\boldsymbol{\nu}^{1}+ (1-\beta)\boldsymbol{\nu}^{2},\theta,w^{tot})\right\}$ $\displaystyle=\widetilde{{\mathcal{L}}}(\beta\boldsymbol{\nu}^{1}+(1-\beta) \boldsymbol{\nu}^{2},\theta).$ $$

The last inequality directly confirms Eq. 16, implying the convexity of $\widetilde{{\mathcal{L}}}(\boldsymbol{\nu},\theta)$ in $\boldsymbol{\nu}$, as desired.
∎

###### Proof.

###### Lemma A.1 .

###### Proof.

### A.3 Proof of Proposition 4.3

Proposition.
Let $\pi^{*}$ be the optimal solution to 4.3. Then $\bm{\pi}^{*}_{tot}(\mathbf{a}|\mathbf{s})=\prod_{i\in\mathcal{N}}\pi^{*}_{i}(a
_{i}|o_{i})$ is also optimal for the global weighted BC problem 8.

###### Proof.

To prove that $\bm{\pi}^{*}_{tot}(\mathbf{a}|\mathbf{s})=\prod_{i\in\mathcal{N}}\pi^{*}_{i}(a
_{i}|o_{i})$ is optimal for the global WBC problem 8, we need to verify that

$$ $\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[w^{tot*}(\textbf {s},\textbf{a})\log\boldsymbol{\pi}_{tot}(\textbf{a}|\textbf{s})\right]\leq \mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[w^{tot*}(\textbf {s},\textbf{a})\log\boldsymbol{\pi}^{*}_{tot}(\textbf{a}|\textbf{s})\right]$ $$

for any global policy $\boldsymbol{\pi}_{tot}\in\Pi_{tot}$.

Since $\boldsymbol{\pi}_{tot}$ is decomposable, there exist local policies $\pi_{i}$ such that

$$ $\boldsymbol{\pi}_{tot}(\textbf{a}|\textbf{s})=\prod_{i\in{\mathcal{N}}}\pi_{i} (a_{i}|o_{i}).$ $$

As a result, we have the following inequalities:

$$ $\displaystyle\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[w^{ tot*}(\textbf{s},\textbf{a})\log\boldsymbol{\pi}_{tot}(\textbf{a}|\textbf{s})\right]$ $\displaystyle=\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[w^ {tot*}(\textbf{s},\textbf{a})\sum_{i\in{\mathcal{N}}}\log\pi_{i}(a_{i}|o_{i})\right]$ $\displaystyle=\sum_{i\in{\mathcal{N}}}\mathbb{E}_{(\textbf{s},\textbf{a})\sim \rho^{\mu_{tot}}}\left[w^{tot*}(\textbf{s},\textbf{a})\log\pi_{i}(a_{i}|o_{i})\right]$ $\displaystyle\leq\sum_{i\in{\mathcal{N}}}\max_{\pi^{\prime}_{i}}\mathbb{E}_{( \textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[w^{tot*}(\textbf{s},\textbf{a })\log\pi^{\prime}_{i}(a_{i}|o_{i})\right]$ $\displaystyle=\sum_{i\in{\mathcal{N}}}\mathbb{E}_{(\textbf{s},\textbf{a})\sim \rho^{\mu_{tot}}}\left[w^{tot*}(\textbf{s},\textbf{a})\log\pi^{*}_{i}(a_{i}|o_ {i})\right]$ $\displaystyle=\mathbb{E}_{(\textbf{s},\textbf{a})\sim\rho^{\mu_{tot}}}\left[w^ {tot*}(\textbf{s},\textbf{a})\log\boldsymbol{\pi}^{*}_{tot}(\textbf{a}|\textbf {s})\right],$ $$

which directly implies that $\boldsymbol{\pi}^{*}_{tot}$ is optimal for the global WBC problem 8.

∎

###### Proof.

## Appendix B Additional Details

### B.1 Offline Multi-Agent Datasets

**Table 3: Overview of datasets used in experiments, including details of trajectories, samples, agent counts, and state, observation, and action space dimensions across SMACv1, SMACv2, and MaMujoco environments, with average returns indicating performance levels.**
| Instances | Trajectories | Samples | Agents | State | Obs | Action | Average |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| dim | dim | dim | returns |  |  |  |  |  |
| 2c_vs_64zg | poor | 0.3K | 21.7K | 2 | 675 | 478 | 70 | 8.9±1.0 |
| medium | 1.0K | 75.9K | 2 | 675 | 478 | 70 | 13.0±1.4 |  |
| good | 1.0K | 118.4K | 2 | 675 | 478 | 70 | 19.9±1.3 |  |
| 5m_vs_6m | poor | 1.0K | 113.7K | 5 | 156 | 124 | 12 | 8.5±1.2 |
| medium | 1.0K | 138.6K | 5 | 156 | 124 | 12 | 11.0±0.6 |  |
| good | 1.0K | 138.7K | 5 | 156 | 124 | 12 | 20.0±0.0 |  |
| 6h_vs_8z | poor | 1.0K | 145.5K | 6 | 213 | 172 | 14 | 9.1±0.8 |
| medium | 1.0K | 177.1K | 6 | 213 | 172 | 14 | 12.0±1.3 |  |
| good | 1.0K | 228.2K | 6 | 213 | 172 | 14 | 17.8±2.1 |  |
| corridor | poor | 1.0K | 307.6K | 6 | 435 | 346 | 30 | 4.9±1.7 |
| medium | 1.0K | 756.1K | 6 | 435 | 346 | 30 | 13.1±1.3 |  |
| good | 1.0K | 601.0K | 6 | 435 | 346 | 30 | 19.9±1.0 |  |
| Protoss | 5_vs_5 | 1.0K | 60.8K | 5 | 130 | 92 | 11 | 16.8±6.3 |
| 10_vs_10 | 1.0K | 68.3K | 10 | 310 | 182 | 16 | 15.7±5.2 |  |
| 10_vs_11 | 1.0K | 62.9K | 10 | 327 | 191 | 17 | 15.3±5.7 |  |
| 20_vs_20 | 1.0K | 76.7K | 20 | 820 | 362 | 26 | 16.2±4.7 |  |
| 20_vs_23 | 1.0K | 65.0K | 20 | 901 | 389 | 29 | 14.0±4.5 |  |
| Terran | 5_vs_5 | 1.0K | 47.6K | 5 | 120 | 82 | 11 | 15.2±7.2 |
| 10_vs_10 | 1.0K | 56.4K | 10 | 290 | 162 | 16 | 14.7±6.2 |  |
| 10_vs_11 | 1.0K | 52.5K | 10 | 306 | 170 | 17 | 12.1±5.7 |  |
| 20_vs_20 | 1.0K | 63.0K | 20 | 780 | 322 | 26 | 14.0±6.0 |  |
| 20_vs_23 | 1.0K | 51.3K | 20 | 858 | 346 | 29 | 11.7±5.7 |  |
| Zerg | 5_vs_5 | 1.0K | 27.5K | 5 | 120 | 82 | 11 | 10.4±5.0 |
| 10_vs_10 | 1.0K | 31.9K | 10 | 290 | 162 | 16 | 14.7±6.0 |  |
| 10_vs_11 | 1.0K | 30.9K | 10 | 306 | 170 | 17 | 12.0±5.1 |  |
| 20_vs_20 | 1.0K | 35.4K | 20 | 780 | 322 | 26 | 12.3±4.2 |  |
| 20_vs_23 | 1.0K | 32.8K | 20 | 858 | 346 | 29 | 10.8±4.0 |  |
| Hopper | expert | 1.5K | 999K | 3 | 42 | 14 | 1 | 2452.0±1097.9 |
| medium | 4.0K | 915K | 3 | 42 | 14 | 1 | 723.6±211.7 |  |
| m-replay | 4.2K | 1311K | 3 | 42 | 14 | 1 | 746.4±671.9 |  |
| m-expert | 5.5K | 1914K | 3 | 42 | 14 | 1 | 1190.6±973.4 |  |
| Ant | expert | 1.0K | 1000K | 2 | 226 | 113 | 4 | 2055.1±22.1 |
| medium | 1.0K | 1000K | 2 | 226 | 113 | 4 | 1418.7±37.0 |  |
| m-replay | 1.8K | 1750K | 2 | 226 | 113 | 4 | 1029.5±141.3 |  |
| m-expert | 2.0K | 2000K | 2 | 226 | 113 | 4 | 1736.9±319.6 |  |
| Half<br>Cheetah | expert | 1.0K | 1000K | 6 | 138 | 23 | 1 | 2785.1±1053.1 |
| medium | 1.0K | 1000K | 6 | 138 | 23 | 1 | 1425.7±520.1 |  |
| m-replay | 1.0K | 1000K | 6 | 138 | 23 | 1 | 655.8±590.4 |  |
| m-expert | 2.0K | 2000K | 6 | 138 | 23 | 1 | 2105.4±1073.2 |  |

### B.2 Implementation Details

Our experiments were implemented using PyTorch and executed in parallel on a single NVIDIA® H100 NVL Tensor Core GPU. Our study required running a large number of sub-tasks, specifically 1,365 in total (i.e., 39 instances across 7 algorithms with 5 different random seeds each).

Figure: Figure 3: Our ComaDICE model architecture.
Refer to caption: x23.png

The offline datasets for each instance are substantial, reaching sizes of up to 7.4 GB. To manage this, we developed a preprocessing step designed to optimize data handling and improve computational efficiency. This process involves reading all transitions from each dataset and combining individual trajectory files into a single large NumPy object that contains batches of trajectories. In this step, we define the data type for each element, such as states (float32), actions (int64), and dones (bool), ensuring consistent and efficient data storage. The processed data is then saved into a compressed NumPy file, which significantly boosts computing performance.

Despite these optimizations, loading the entire dataset still requires a large amount of RAM. By leveraging parallel processing and efficient data management strategies, we effectively managed the extensive computational and memory demands of our experiments. This approach allowed us to handle the large-scale data and complex computations necessary for our study.

#### B.2.1 Hyper-parameters

**Table 4: Hyperparameters for our algorithm**
| Hyperparameter | Value |
| --- | --- |
| Optimizer | Adam |
| Learning rate (Q-value and policy networks) | $1\times 10^{-4}$ |
| Tau ($\tau$) | 0.005 |
| Gamma ($\gamma$) | 0.99 |
| Batch size | 128 |
| Agent hidden dimension | 256 |
| Mixer hidden dimension | 64 |
| Number of seeds | 5 |
| Number of episodes per evaluation step | 32 |
| Number of evaluation steps | 100 |
| Lambda scale ($\lambda$) | 1.0 |
| Alpha ($\alpha$) | 10 |
| f-divergence | soft-$\chi^{2}$ |

In our study, we developed two versions of our algorithm: a continuous version for MaMujoco using Gaussian distributions (torch.distributions.Normal), and a discrete version for SMACv1 and SMACv2 using Categorical distributions (torch.distributions.Categorical). In the discrete setting, action probabilities are computed using softmax over available actions only, ensuring zero probability for unavailable actions, which enhances the accuracy of log likelihood calculations. Key hyperparameters are listed on the Table 4. Experiments were conducted with 5 seeds, 32 episodes per evaluation step, and 100 evaluation steps.

### B.3 Additional Experimental Details

We evaluate the performance of our ComaDICE algorithm using two key metrics: mean and standard deviation (std) of returns and winrates. Returns measure the average rewards accumulated by agents, calculated across five random seeds to ensure robustness, while winrates, applicable only to competitive environments like SMACv1 and SMACv2, indicate the success rate against other agents. For cooperative settings such as MaMujoco, winrates are not applicable. We also include figures showing evaluation curves, highlighting how each method’s performance evolves during training with offline datasets. These metrics and visualizations provide a comprehensive overview of our algorithm’s effectiveness and consistency in various MARL tasks.

#### B.3.1 Returns

Tables 5, 6, and 7 present the returns from our experimental results across the SMACv1, SMACv2, and Multi-Agent MuJoCo environments, highlighting the performance of our proposed algorithm, ComaDICE, alongside baseline methods such as BC, BCQ, CQL, ICQ, OMAR, and OMIGA. Our results demonstrate that ComaDICE consistently achieves superior returns, particularly excelling in more complex difficulty tasks. Figures 4, 5, and 6 illustrate the learning curves for these algorithms, showing that ComaDICE not only outperforms other algorithms in terms of mean returns but also exhibits lower standard deviation, indicating robust and stable performance. This suggests that ComaDICE effectively handles distributional shifts in offline settings. These findings underscore our algorithm’s adaptability and effectiveness in diverse multi-agent coordination scenarios, setting a new benchmark in offline MARL.

**Table 5: Comparison of average returns for ComaDICE and baselines on SMACv1 benchmarks.**
| Instances | BC | BCQ | CQL | ICQ | OMAR | OMIGA | ComaDICE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2c_vs_64zg | poor | 11.6±0.4 | 12.5±0.2 | 10.8±0.5 | 12.6±0.2 | 11.3±0.5 | 13.0±0.7 | 12.1±0.5 |
| medium | 13.4±1.9 | 15.6±0.4 | 12.8±1.6 | 15.6±0.6 | 10.2±0.2 | 16.0±0.2 | 16.3±0.7 |  |
| good | 17.9±1.3 | 19.1±0.3 | 18.5±1.0 | 18.8±0.2 | 17.3±0.8 | 19.1±0.3 | 20.3±0.1 |  |
| 5m_vs_6m | poor | 7.0±0.5 | 7.6±0.4 | 7.4±0.1 | 7.3±0.2 | 7.3±0.4 | 7.5±0.2 | 8.1±0.5 |
| medium | 7.0±0.8 | 7.6±0.1 | 7.8±0.1 | 7.8±0.3 | 7.1±0.5 | 7.9±0.6 | 8.7±0.4 |  |
| good | 7.0±0.5 | 7.8±0.1 | 8.1±0.2 | 7.9±0.3 | 7.4±0.6 | 8.3±0.4 | 8.7±0.5 |  |
| 6h_vs_8z | poor | 8.6±0.8 | 10.8±0.2 | 10.8±0.5 | 10.6±0.1 | 10.6±0.2 | 11.3±0.2 | 11.4±0.6 |
| medium | 9.5±0.3 | 11.8±0.2 | 11.3±0.3 | 11.1±0.3 | 10.4±0.2 | 12.2±0.2 | 12.8±0.2 |  |
| good | 10.0±1.7 | 12.2±0.2 | 10.4±0.2 | 11.8±0.1 | 9.9±0.3 | 12.5±0.2 | 13.1±0.5 |  |
| corridor | poor | 2.9±0.6 | 4.5±0.9 | 4.1±0.6 | 4.5±0.3 | 4.3±0.5 | 5.6±0.3 | 6.4±0.5 |
| medium | 7.4±0.8 | 10.8±0.9 | 7.0±0.7 | 11.3±1.6 | 7.3±0.7 | 11.7±1.3 | 12.9±0.6 |  |
| good | 10.8±2.6 | 15.2±1.2 | 5.2±0.8 | 15.5±1.1 | 6.7±0.7 | 15.9±0.9 | 18.0±0.1 |  |

Figure: Figure 4: Evaluation of SMACv1 tasks comparing the returns achieved by ComaDICE and baselines.
Refer to caption: x24.png

**Table 6: Comparison of average returns for ComaDICE and baselines on SMACv2 tasks.**
| Instances | BC | BCQ | CQL | ICQ | OMAR | OMIGA | ComaDICE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Protoss | 5_vs_5 | 13.2±0.7 | 6.8±1.6 | 9.3±1.6 | 10.7±1.2 | 8.9±0.8 | 14.3±1.4 | 14.4±1.1 |
| 10_vs_10 | 12.0±1.9 | 7.7±1.3 | 11.3±0.9 | 10.4±1.6 | 8.8±0.6 | 14.2±1.5 | 14.6±1.8 |  |
| 10_vs_11 | 11.2±0.5 | 5.2±1.4 | 7.9±0.8 | 10.3±0.7 | 8.0±0.3 | 12.1±0.5 | 13.2±0.9 |  |
| 20_vs_20 | 13.1±0.5 | 4.8±0.6 | 10.5±0.9 | 11.8±0.5 | 9.1±0.5 | 14.0±0.9 | 14.8±1.0 |  |
| 20_vs_23 | 11.2±0.5 | 3.5±0.6 | 5.6±0.7 | 10.2±0.7 | 7.4±0.7 | 13.0±1.1 | 13.3±0.9 |  |
| Terran | 5_vs_5 | 10.8±1.4 | 6.4±1.1 | 6.5±0.9 | 6.8±0.6 | 6.9±0.6 | 10.5±1.2 | 10.7±1.5 |
| 10_vs_10 | 10.3±0.3 | 4.6±0.4 | 6.8±0.6 | 8.7±1.4 | 7.6±1.0 | 10.1±0.6 | 11.8±0.9 |  |
| 10_vs_11 | 9.0±0.7 | 3.6±1.1 | 5.5±0.2 | 5.5±0.9 | 5.9±0.7 | 8.8±1.4 | 9.4±0.9 |  |
| 20_vs_20 | 10.8±0.8 | 3.9±0.6 | 4.3±0.6 | 8.3±0.3 | 7.3±0.4 | 10.5±0.7 | 11.8±0.5 |  |
| 20_vs_23 | 7.2±1.0 | 1.2±1.0 | 1.6±0.2 | 5.3±0.5 | 5.1±0.3 | 7.9±0.6 | 8.2±0.7 |  |
| Zerg | 5_vs_5 | 10.5±2.2 | 6.6±0.2 | 6.7±0.5 | 6.5±0.9 | 7.7±0.9 | 8.9±1.1 | 10.7±2.0 |
| 10_vs_10 | 11.0±0.8 | 7.3±1.0 | 7.2±0.3 | 7.7±1.1 | 7.5±0.8 | 11.8±1.6 | 11.5±1.0 |  |
| 10_vs_11 | 9.2±1.1 | 7.6±0.9 | 6.7±0.4 | 6.8±1.0 | 6.5±1.0 | 9.5±1.2 | 11.0±0.9 |  |
| 20_vs_20 | 9.3±0.5 | 3.7±0.4 | 4.7±0.3 | 6.9±0.5 | 6.9±0.8 | 9.2±0.5 | 9.4±1.2 |  |
| 20_vs_23 | 8.5±0.7 | 3.3±0.3 | 4.1±0.6 | 6.9±0.5 | 5.7±0.4 | 9.8±0.6 | 10.5±0.8 |  |

Figure: Figure 5: Evaluation of SMACv2 tasks comparing the returns achieved by ComaDICE and baselines.
Refer to caption: x24.png

**Table 7: Comparison of average returns for ComaDICE and baselines on MaMujoco benchmarks.**
| Instances | expert | medium | m-replay | m-expert |  |
| --- | --- | --- | --- | --- | --- |
| Hopper | BC | 209.8±191.1 | 511.9±7.4 | 133.3±53.5 | 155.3±111.5 |
| BCQ | 77.9±58.0 | 44.6±20.6 | 26.5±24.0 | 54.3±23.7 |  |
| CQL | 159.1±313.8 | 401.3±199.9 | 31.4±15.2 | 64.8±123.3 |  |
| ICQ | 754.7±806.3 | 501.8±14.0 | 195.4±103.6 | 355.4±373.9 |  |
| OMAR | 2.4±1.5 | 21.3±24.9 | 3.3±3.2 | 1.4±0.9 |  |
| OMIGA | 859.6±709.5 | 1189.3±544.3 | 774.2±494.3 | 709.0±595.7 |  |
| ComaDICE | 2827.7±62.9 | 822.6±66.2 | 906.3±242.1 | 1362.4±522.9 |  |
| Ant | BC | 2046.3±6.2 | 1421.1±7.9 | 994.0±20.3 | 1561.7±64.8 |
| BCQ | 1317.7±286.3 | 1059.6±91.2 | 950.8±48.8 | 1020.9±242.7 |  |
| CQL | 1042.4±2021.6 | 533.9±1766.4 | 234.6±1618.3 | 800.2±1621.5 |  |
| ICQ | 2050.0±11.9 | 1412.4±10.9 | 1016.7±53.5 | 1590.2±85.6 |  |
| OMAR | 312.5±297.5 | -1710.0±1589.0 | -2014.2±844.7 | -2992.8±7.0 |  |
| OMIGA | 2055.5±1.6 | 1418.4±5.4 | 1105.1±88.9 | 1720.3±110.6 |  |
| ComaDICE | 2056.9±5.9 | 1425.0±2.9 | 1122.9±61.0 | 1813.9±68.4 |  |
| Half<br>Cheetah | BC | 3251.2±386.8 | 2280.3±178.2 | 1886.2±390.8 | 2451.9±783.0 |
| BCQ | 2992.7±629.7 | 2590.5±1110.4 | -333.6±152.1 | 3543.7±780.9 |  |
| CQL | 1189.5±1034.5 | 1011.3±1016.9 | 1998.7±693.9 | 1194.2±1081.0 |  |
| ICQ | 2955.9±459.2 | 2549.3±96.3 | 1922.4±612.9 | 2834.0±420.3 |  |
| OMAR | -206.7±161.1 | -265.7±147.0 | -235.4±154.9 | -253.8±63.9 |  |
| OMIGA | 3383.6±552.7 | 3608.1±237.4 | 2504.7±83.5 | 2948.5±518.9 |  |
| ComaDICE | 4082.9±45.7 | 2664.7±54.2 | 2855.0±242.2 | 3889.7±81.6 |  |

Figure: Figure 6: Evaluation of MaMujoco tasks comparing the returns achieved by ComaDICE and baselines.
Refer to caption: x24.png

#### B.3.2 Winrates

In this section, we analyze the winrates of our ComaDICE algorithm across various multi-agent reinforcement learning scenarios. Winrates are crucial in competitive environments like SMACv1 and SMACv2, as they measure the algorithm’s success against other agents. Our results demonstrate that ComaDICE consistently achieves higher winrates compared to baseline methods. Notably, ComaDICE performs well across both simple and complex tasks, reflecting its robustness and adaptability. As shown in Tables 8 and 9, as well as Figures 7 and 8, ComaDICE not only excels in average winrates but also exhibits lower variance, indicating stable performance across different trials. These findings highlight ComaDICE’s ability to effectively manage distributional shifts.

**Table 8: Comparison of average winrates for ComaDICE and baselines on SMACv1 benchmarks.**
| Instances | BC | BCQ | CQL | ICQ | OMAR | OMIGA | ComaDICE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2c_vs_64zg | poor | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.6±1.3 |
| medium | 1.9±1.5 | 2.5±3.6 | 2.5±3.6 | 1.9±1.5 | 1.2±1.5 | 6.2±5.6 | 8.8±7.0 |  |
| good | 31.2±9.9 | 35.6±8.8 | 44.4±13.0 | 28.7±4.6 | 28.7±9.1 | 40.6±9.5 | 55.0±1.5 |  |
| 5m_vs_6m | poor | 2.5±1.3 | 1.2±1.5 | 1.2±1.5 | 1.2±1.5 | 0.6±1.2 | 6.9±1.2 | 4.4±4.2 |
| medium | 1.9±1.5 | 1.2±1.5 | 2.5±1.2 | 1.2±1.5 | 0.6±1.2 | 2.5±3.1 | 7.5±2.5 |  |
| good | 2.5±2.3 | 1.9±2.5 | 1.9±1.5 | 3.8±2.3 | 3.8±1.2 | 6.9±1.2 | 8.1±3.2 |  |
| 6h_vs_8z | poor | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 1.9±3.8 |
| medium | 1.9±1.5 | 1.9±1.5 | 1.9±1.5 | 2.5±1.2 | 1.9±1.5 | 1.2±1.5 | 3.1±2.0 |  |
| good | 8.8±1.2 | 8.8±3.6 | 7.5±1.5 | 9.4±2.0 | 0.6±1.3 | 5.6±3.6 | 11.2±5.4 |  |
| corridor | poor | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.6±1.3 | 0.0±0.0 | 0.0±0.0 | 0.6±1.3 |
| medium | 15.0±2.3 | 23.1±1.5 | 14.4±1.5 | 22.5±3.1 | 11.9±2.3 | 23.8±5.1 | 27.3±3.4 |  |
| good | 30.6±4.1 | 42.5±6.4 | 5.6±1.2 | 42.5±6.4 | 3.1±0.0 | 41.9±6.4 | 48.8±2.5 |  |

Figure: Figure 7: Evaluation of SMACv1 tasks comparing the winrates achieved by ComaDICE and baselines.
Refer to caption: x24.png

**Table 9: Comparison of average winrates for ComaDICE and baselines on SMACv2 benchmarks.**
| Instances | BC | BCQ | CQL | ICQ | OMAR | OMIGA | ComaDICE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Protoss | 5_vs_5 | 36.9±8.7 | 16.2±2.3 | 10.0±4.1 | 36.9±9.1 | 21.2±4.1 | 33.1±5.4 | 46.2±6.1 |
| 10_vs_10 | 36.2±10.6 | 9.4±5.6 | 26.2±7.6 | 28.1±6.6 | 13.8±7.0 | 40.0±10.7 | 50.6±8.7 |  |
| 10_vs_11 | 19.4±4.6 | 10.0±4.1 | 10.6±5.4 | 12.5±4.4 | 12.5±3.4 | 16.2±6.1 | 20.0±4.2 |  |
| 20_vs_20 | 37.5±4.4 | 6.2±2.0 | 11.9±4.1 | 32.5±8.1 | 23.8±2.5 | 36.2±5.1 | 47.5±7.8 |  |
| 20_vs_23 | 13.8±1.5 | 1.2±1.5 | 0.0±0.0 | 12.5±5.6 | 11.2±7.8 | 12.5±8.1 | 13.8±5.8 |  |
| Terran | 5_vs_5 | 30.0±4.2 | 12.5±6.2 | 9.4±7.9 | 23.1±5.8 | 14.4±4.7 | 28.1±4.4 | 30.6±8.2 |
| 10_vs_10 | 29.4±5.8 | 6.9±6.1 | 9.4±5.6 | 16.9±5.8 | 15.0±4.6 | 29.4±3.2 | 32.5±5.8 |  |
| 10_vs_11 | 16.2±3.6 | 3.8±4.6 | 7.5±6.4 | 5.0±4.2 | 9.4±5.6 | 12.5±5.2 | 19.4±5.4 |  |
| 20_vs_20 | 26.2±10.4 | 5.0±3.2 | 10.6±4.2 | 15.6±3.4 | 7.5±7.3 | 21.9±4.4 | 29.4±3.8 |  |
| 20_vs_23 | 4.4±4.2 | 0.0±0.0 | 0.0±0.0 | 7.5±6.1 | 5.0±4.2 | 4.4±2.5 | 9.4±5.2 |  |
| Zerg | 5_vs_5 | 26.9±10.0 | 14.4±4.2 | 14.4±5.8 | 18.8±7.1 | 13.8±6.1 | 21.9±5.9 | 31.2±7.7 |
| 10_vs_10 | 25.0±2.8 | 5.6±4.6 | 5.6±4.6 | 15.6±7.4 | 19.4±2.3 | 23.8±6.4 | 33.8±11.8 |  |
| 10_vs_11 | 13.8±4.7 | 9.4±5.2 | 6.2±4.4 | 10.6±6.7 | 10.6±3.8 | 13.8±6.7 | 19.4±3.6 |  |
| 20_vs_20 | 8.1±1.5 | 2.5±1.2 | 1.2±1.5 | 10.0±7.8 | 12.5±4.4 | 10.0±2.3 | 9.4±6.2 |  |
| 20_vs_23 | 7.5±3.2 | 0.6±1.3 | 1.2±1.5 | 7.5±3.2 | 3.8±2.3 | 4.4±4.2 | 11.2±4.2 |  |

Figure: Figure 8: Evaluation of SMACv2 tasks comparing the winrates achieved by ComaDICE and baselines.
Refer to caption: x24.png

### B.4 Ablation Study: Different Values of Alpha

We provide more experimental details for ablation study assessing the impact of varying the regularization parameter alpha ($\alpha$) on the performance of our ComaDICE.

#### B.4.1 Returns

Our results, in Tables 10, 11, and 12, show that the performance of ComaDICE is sensitive to the choice of $\alpha$. Lower values of $\alpha$ tend to prioritize imitation learning, leading to suboptimal performance in terms of returns, whereas higher values facilitate better adaptation to the offline data, achieving superior returns. Notably, an $\alpha$ value of 10 consistently yielded the best results across most tasks, indicating an optimal balance between exploration and exploitation in offline settings. This ablation study underscores the importance of selecting an appropriate $\alpha$ to enhance the algorithm’s robustness and effectiveness in handling distributional shifts in offline multi-agent reinforcement learning scenarios.

**Table 10: Impact of alpha on returns for ComaDICE and baselines in SMACv1.**
| Instances | $\alpha=0.01$ | $\alpha=0.1$ | $\alpha=1$ | $\alpha=10$ | $\alpha=100$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| 2c_vs_64zg | poor | 10.6±0.5 | 11.1±0.4 | 11.1±0.1 | 12.1±0.5 | 11.8±0.2 |
| medium | 9.6±0.5 | 13.1±0.8 | 12.5±2.4 | 16.3±0.7 | 16.0±0.3 |  |
| good | 11.1±1.4 | 9.6±2.7 | 17.4±0.5 | 20.3±0.1 | 19.9±0.1 |  |
| 5m_vs_6m | poor | 5.7±0.1 | 5.1±0.3 | 7.1±0.7 | 8.1±0.5 | 7.7±0.3 |
| medium | 5.6±0.1 | 5.3±0.2 | 7.8±0.8 | 8.7±0.4 | 8.5±0.7 |  |
| good | 5.7±0.1 | 5.7±0.2 | 7.8±0.5 | 8.7±0.5 | 8.8±0.8 |  |
| 6h_vs_8z | poor | 8.5±0.2 | 9.6±0.3 | 10.0±0.3 | 11.4±0.6 | 10.7±0.4 |
| medium | 8.5±0.6 | 10.5±0.8 | 10.7±0.5 | 12.8±0.2 | 12.3±0.3 |  |
| good | 7.9±0.1 | 9.5±0.6 | 11.3±0.6 | 13.1±0.5 | 12.8±0.4 |  |
| corridor | poor | 2.1±0.4 | 3.7±1.0 | 6.1±0.8 | 6.4±0.5 | 5.0±1.1 |
| medium | 1.7±1.0 | 2.2±1.7 | 11.3±0.3 | 12.9±0.6 | 13.3±0.1 |  |
| good | 4.7±2.4 | 3.8±5.0 | 15.7±0.3 | 18.0±0.1 | 17.4±0.1 |  |

**Table 11: Impact of alpha on returns for ComaDICE and baselines in SMACv2.**
| Instances | $\alpha=0.01$ | $\alpha=0.1$ | $\alpha=1$ | $\alpha=10$ | $\alpha=100$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Protoss | 5_vs_5 | 12.2±1.0 | 13.1±1.3 | 13.2±1.1 | 14.4±1.1 | 14.0±2.0 |
| 10_vs_10 | 12.8±0.9 | 14.0±0.8 | 13.4±1.2 | 14.6±1.8 | 14.1±1.3 |  |
| 10_vs_11 | 9.9±1.1 | 11.1±0.8 | 11.3±1.2 | 13.2±0.9 | 12.2±1.1 |  |
| 20_vs_20 | 10.3±0.5 | 11.1±1.0 | 12.2±0.9 | 14.8±1.0 | 13.2±0.4 |  |
| 20_vs_23 | 8.0±2.3 | 11.2±1.2 | 11.7±0.6 | 13.3±0.9 | 13.2±0.5 |  |
| Terran | 5_vs_5 | 11.1±1.8 | 10.1±1.2 | 9.0±1.0 | 10.7±1.5 | 12.6±1.9 |
| 10_vs_10 | 8.5±0.8 | 10.3±0.7 | 10.4±1.1 | 11.8±0.9 | 11.8±1.7 |  |
| 10_vs_11 | 7.5±0.7 | 8.6±2.1 | 8.5±1.6 | 9.4±0.9 | 9.6±0.9 |  |
| 20_vs_20 | 6.2±1.1 | 6.4±1.7 | 9.1±0.7 | 11.8±0.5 | 9.3±0.6 |  |
| 20_vs_23 | 5.5±1.1 | 6.5±1.6 | 6.5±0.8 | 8.2±0.7 | 8.2±0.4 |  |
| Zerg | 5_vs_5 | 7.9±0.6 | 9.3±0.9 | 10.5±1.4 | 10.7±2.0 | 10.4±1.2 |
| 10_vs_10 | 10.9±1.5 | 11.4±1.5 | 11.8±0.7 | 11.5±1.0 | 10.9±2.2 |  |
| 10_vs_11 | 10.1±2.5 | 9.1±1.2 | 10.0±1.2 | 11.0±0.9 | 9.8±0.8 |  |
| 20_vs_20 | 8.0±0.5 | 9.2±1.3 | 9.2±1.0 | 9.4±1.2 | 10.5±0.9 |  |
| 20_vs_23 | 9.1±1.1 | 10.0±0.7 | 10.4±0.6 | 10.5±0.8 | 10.1±0.7 |  |

**Table 12: Impact of alpha on returns for ComaDICE and baselines in MaMujoco.**
| Instances | $\alpha=0.01$ | $\alpha=0.1$ | $\alpha=1$ | $\alpha=10$ | $\alpha=100$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Hopper | expert | 147.3±67.9 | 107.9±65.5 | 545.7±820.6 | 2827.7±62.9 | 2690.7±58.6 |
| medium | 149.6±96.8 | 107.5±66.9 | 244.7±267.5 | 822.6±66.2 | 807.5±122.2 |  |
| m-replay | 165.6±104.1 | 109.6±38.7 | 155.6±61.6 | 906.3±242.1 | 186.5±16.8 |  |
| m-expert | 119.1±77.1 | 95.6±69.5 | 58.8±26.1 | 1362.4±522.9 | 1358.4±595.1 |  |
| Ant | expert | 1016.4±196.5 | 1179.0±273.7 | 1927.7±174.1 | 2056.9±5.9 | 1950.0±3.3 |
| medium | 907.3±32.2 | 1000.0±90.4 | 1424.3±3.1 | 1425.0±2.9 | 1354.6±2.5 |  |
| m-replay | 969.1±21.9 | 978.4±39.6 | 944.6±28.9 | 1122.9±61.0 | 1072.1±41.4 |  |
| m-expert | 915.8±364.1 | 1132.9±282.2 | 738.5±250.2 | 1813.9±68.4 | 1559.6±86.8 |  |
| Half<br>Cheetah | expert | 1068.9±635.2 | 935.2±905.9 | 3637.0±80.9 | 4082.9±45.7 | 3843.7±149.4 |
| medium | 575.9±724.8 | 445.2±403.9 | 2690.0±92.4 | 2664.7±54.2 | 2523.4±59.0 |  |
| m-replay | 412.3±310.5 | 233.5±270.1 | 861.6±173.5 | 2855.0±242.2 | 2557.4±241.5 |  |
| m-expert | -107.5±298.1 | -275.9±544.5 | 1136.9±1608.3 | 3889.7±81.6 | 3605.6±70.4 |  |

Figure: Figure 9: Impact of alpha on returns for ComaDICE and baselines.
Refer to caption: x91.png

#### B.4.2 Winrates

In the A.4.2 section of the appendix, we investigate the impact of varying $\alpha$ on winrates across different multi-agent reinforcement learning environments. We observe that an intermediate $\alpha$ value of 10 consistently yields optimal results, suggesting it strikes an effective balance between conservative policy adherence and exploration of the offline dataset. This section underscores the importance of fine-tuning $\alpha$ to enhance the robustness and efficacy of the ComaDICE algorithm in managing distributional shifts within competitive multi-agent settings.

**Table 13: Impact of alpha on winrates for ComaDICE and baselines in SMACv1.**
| Instances | $\alpha=0.01$ | $\alpha=0.1$ | $\alpha=1$ | $\alpha=10$ | $\alpha=100$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| 2c_vs_64zg | poor | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.6±1.3 | 0.6±1.3 |
| medium | 0.0±0.0 | 1.9±3.8 | 5.0±5.1 | 8.8±7.0 | 8.8±4.6 |  |
| good | 0.6±1.2 | 0.0±0.0 | 40.6±4.0 | 55.0±1.5 | 51.9±1.5 |  |
| 5m_vs_6m | poor | 0.0±0.0 | 0.0±0.0 | 4.4±4.7 | 4.4±4.2 | 1.9±1.5 |
| medium | 0.0±0.0 | 0.0±0.0 | 8.1±6.4 | 7.5±2.5 | 7.5±3.8 |  |
| good | 0.0±0.0 | 0.0±0.0 | 6.2±4.4 | 8.1±3.2 | 10.0±6.1 |  |
| 6h_vs_8z | poor | 0.0±0.0 | 0.0±0.0 | 1.9±3.8 | 1.9±3.8 | 0.6±1.3 |
| medium | 0.0±0.0 | 0.6±1.3 | 1.9±1.5 | 3.1±2.0 | 3.1±2.0 |  |
| good | 0.0±0.0 | 0.0±0.0 | 7.5±5.8 | 11.2±5.4 | 7.5±7.3 |  |
| corridor | poor | 0.0±0.0 | 0.6±1.2 | 0.0±0.0 | 0.6±1.3 | 1.2±1.5 |
| medium | 0.0±0.0 | 0.0±0.0 | 30.0±5.1 | 27.3±3.4 | 34.4±2.8 |  |
| good | 0.0±0.0 | 4.4±8.8 | 48.8±4.7 | 48.8±2.5 | 49.4±3.6 |  |

**Table 14: Impact of alpha on winrates for ComaDICE and baselines in SMACv2.**
| Instances | $\alpha=0.01$ | $\alpha=0.1$ | $\alpha=1$ | $\alpha=10$ | $\alpha=100$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Protoss | 5_vs_5 | 20.6±10.0 | 31.9±6.1 | 50.0±2.8 | 46.2±6.1 | 46.2±8.5 |
| 10_vs_10 | 19.4±6.1 | 25.0±3.4 | 45.0±11.1 | 50.6±8.7 | 51.2±7.6 |  |
| 10_vs_11 | 0.0±0.0 | 6.2±9.7 | 18.8±8.1 | 20.0±4.2 | 29.4±8.3 |  |
| 20_vs_20 | 1.2±1.5 | 8.8±7.8 | 28.1±8.6 | 47.5±7.8 | 40.6±6.2 |  |
| 20_vs_23 | 0.0±0.0 | 1.9±2.5 | 9.4±6.6 | 13.8±5.8 | 17.5±5.1 |  |
| Terran | 5_vs_5 | 25.6±4.6 | 22.5±7.2 | 30.6±4.1 | 30.6±8.2 | 41.2±4.6 |
| 10_vs_10 | 15.0±8.7 | 28.7±7.2 | 33.8±9.4 | 32.5±5.8 | 43.8±7.1 |  |
| 10_vs_11 | 3.8±2.3 | 13.8±9.2 | 14.4±9.2 | 19.4±5.4 | 16.2±10.3 |  |
| 20_vs_20 | 0.6±1.2 | 2.5±3.6 | 18.8±2.0 | 29.4±3.8 | 21.9±3.4 |  |
| 20_vs_23 | 0.6±1.3 | 2.5±3.6 | 2.5±3.6 | 9.4±5.2 | 6.2±2.0 |  |
| Zerg | 5_vs_5 | 10.0±4.6 | 20.0±5.8 | 28.7±4.6 | 31.2±7.7 | 25.0±8.6 |
| 10_vs_10 | 13.8±9.0 | 20.6±8.3 | 29.4±9.0 | 33.8±11.8 | 31.9±6.7 |  |
| 10_vs_11 | 9.4±9.5 | 12.5±6.8 | 16.9±3.2 | 19.4±3.6 | 17.5±9.2 |  |
| 20_vs_20 | 0.0±0.0 | 1.9±1.5 | 6.9±6.1 | 9.4±6.2 | 12.5±4.0 |  |
| 20_vs_23 | 1.2±1.5 | 3.8±2.3 | 12.5±4.0 | 11.2±4.2 | 11.9±6.1 |  |

Figure: Figure 10: Impact of alpha on winrates for ComaDICE and baselines.
Refer to caption: x98.png

### B.5 Ablation Study: Different Forms of f-divergence

We conduct an ablation study to examine the effects of different functions of $f$-divergence on the performance of our ComaDICE algorithm across various multi-agent reinforcement learning environments. The study specifically evaluates three types of $f$-divergence: Kullback-Leibler (KL), $\chi^{2}$, and $\text{Soft-}\chi^{2}$ .

##### KL-Divergence:

This is a well-known measure of how one probability distribution diverges from a second, expected probability distribution. It is defined as:

$$ $f_{\text{KL}}(x)=x\log x-x+1$ $$

The corresponding inverse derivative, which is used in optimization, is:

$$ $(f^{\prime}_{\text{KL}})^{-1}(x)=\exp(x-1)$ $$

KL-divergence can lead to numerical instability due to the exponential function, especially when the values become large.

##### χ 2 superscript 𝜒 2 \chi^{2} italic_χ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT -Divergence:

This divergence measures the difference between two probability distributions by considering the square of the differences. It is expressed as:

$$ $f_{\chi^{2}}(x)=\frac{1}{2}(x-1)^{2}$ $$

The inverse derivative is:

$$ $(f^{\prime}_{\chi^{2}})^{-1}(x)=x+1$ $$

While this function avoids the exponential instability seen in KL-divergence, it may suffer from zero gradients for negative values, which can slow down or halt training.

##### Soft- χ 2 superscript 𝜒 2 \chi^{2} italic_χ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT Divergence:

This function combines the forms of KL and $\chi^{2}$ divergences to mitigate both numerical instability and the dying gradient problem. It is defined piecewise as:

$$ $f_{\text{Soft-}\chi^{2}}(x)=\begin{cases}x\log x-x+1&\text{if }0<x<1\\ \frac{1}{2}(x-1)^{2}&\text{if }x\geq 1\end{cases}$ $$

The inverse derivative is:

$$ $(f^{\prime}_{\text{Soft-}\chi^{2}})^{-1}(x)=\begin{cases}\exp(x)&\text{if }x<0 \\ x+1&\text{if }x\geq 0\end{cases}$ $$

This choice provides a stable optimization process by maintaining non-zero gradients and avoiding large exponential values, making it suitable for reinforcement learning tasks.

We assess their impact on both returns and winrates in environments such as SMACv1, SMACv2, and MaMujoco. Our results, detailed in Tables 15-19, reveal that the choice of $f$-divergence function significantly influences the algorithm’s effectiveness. For instance, the $\text{Soft-}\chi^{2}$ divergence consistently yields superior returns and competitive winrates across most scenarios, suggesting its robustness in managing distributional shifts in offline settings. Conversely, while $\text{Soft-}\chi^{2}$ divergence also performs well, particularly in environments with higher complexity, KL divergence shows varying results, indicating its sensitivity to specific task dynamics. This comprehensive analysis underscores the importance of selecting an appropriate $f$-divergence function to optimize ComaDICE’s performance in diverse multi-agent reinforcement learning contexts.

#### B.5.1 Returns

**Table 15: Impact of $f$-divergence on returns for ComaDICE and baselines in SMACv1.**
| Instances | $f_{\chi^{2}}(x)$ | $f_{\text{KL}}(x)$ | $f_{\text{Soft-}\chi^{2}}(x)$ |  |
| --- | --- | --- | --- | --- |
| 2c_vs_64zg | poor | 11.6±0.2 | 11.1±0.3 | 12.1±0.5 |
| medium | 16.1±0.6 | 15.7±0.3 | 16.3±0.7 |  |
| good | 19.7±0.1 | 19.3±0.1 | 20.3±0.1 |  |
| 5m_vs_6m | poor | 7.8±0.4 | 7.5±0.5 | 8.1±0.5 |
| medium | 8.1±0.5 | 7.7±0.4 | 8.7±0.4 |  |
| good | 8.7±0.6 | 8.1±0.4 | 8.7±0.5 |  |
| 6h_vs_8z | poor | 10.5±0.3 | 10.0±0.2 | 11.4±0.6 |
| medium | 12.9±0.4 | 12.4±0.5 | 12.8±0.2 |  |
| good | 12.7±0.4 | 12.4±0.5 | 13.1±0.5 |  |
| corridor | poor | 6.5±0.5 | 6.1±0.4 | 6.4±0.5 |
| medium | 12.7±0.7 | 12.0±0.7 | 12.9±0.6 |  |
| good | 17.3±0.1 | 16.9±0.1 | 18.0±0.1 |  |

**Table 16: Impact of $f$-divergence on returns for ComaDICE and baselines in SMACv2.**
| Instances | $f_{\chi^{2}}(x)$ | $f_{\text{KL}}(x)$ | $f_{\text{Soft-}\chi^{2}}(x)$ |  |
| --- | --- | --- | --- | --- |
| Protoss | 5_vs_5 | 14.6±0.5 | 13.6±0.9 | 14.4±1.1 |
| 10_vs_10 | 14.7±1.3 | 13.7±1.6 | 14.6±1.8 |  |
| 10_vs_11 | 12.8±1.0 | 11.4±1.7 | 13.2±0.9 |  |
| 20_vs_20 | 12.7±0.3 | 13.1±0.7 | 14.8±1.0 |  |
| 20_vs_23 | 12.4±0.9 | 12.5±0.7 | 13.3±0.9 |  |
| Terran | 5_vs_5 | 11.1±1.2 | 12.7±2.0 | 10.7±1.5 |
| 10_vs_10 | 9.8±0.9 | 10.7±1.3 | 11.8±0.9 |  |
| 10_vs_11 | 8.9±0.8 | 8.9±1.0 | 9.4±0.9 |  |
| 20_vs_20 | 10.5±0.5 | 10.2±0.7 | 11.8±0.5 |  |
| 20_vs_23 | 8.2±0.4 | 7.4±0.7 | 8.2±0.7 |  |
| Zerg | 5_vs_5 | 10.0±0.8 | 9.6±1.5 | 10.7±2.0 |
| 10_vs_10 | 12.4±1.2 | 10.3±1.1 | 11.5±1.0 |  |
| 10_vs_11 | 8.9±0.4 | 9.1±1.1 | 11.0±0.9 |  |
| 20_vs_20 | 9.0±0.8 | 9.0±0.6 | 9.4±1.2 |  |
| 20_vs_23 | 10.2±1.0 | 9.3±0.8 | 10.5±0.8 |  |

**Table 17: Impact of $f$-divergence on returns for ComaDICE and baselines in MaMujoco.**
| Instances | $f_{\chi^{2}}(x)$ | $f_{\text{KL}}(x)$ | $f_{\text{Soft-}\chi^{2}}(x)$ |  |
| --- | --- | --- | --- | --- |
| Hopper | expert | 2625.0±191.3 | 2018.7±972.0 | 2827.7±62.9 |
| medium | 794.4±69.2 | 295.5±227.1 | 822.6±66.2 |  |
| m-replay | 221.3±58.0 | 129.9±55.0 | 906.3±242.1 |  |
| m-expert | 1294.1±520.4 | 105.5±103.9 | 1362.4±522.9 |  |
| Ant | expert | 1945.2±2.8 | 1884.1±27.8 | 2056.9±5.9 |
| medium | 1359.2±3.2 | 1346.2±49.8 | 1425.0±2.9 |  |
| m-replay | 1111.1±57.8 | 987.5±33.9 | 1122.9±61.0 |  |
| m-expert | 1655.9±42.8 | 1182.5±405.1 | 1813.9±68.4 |  |
| Half<br>Cheetah | expert | 3860.6±91.5 | 3830.0±88.8 | 4082.9±45.7 |
| medium | 2532.3±81.9 | 2347.8±171.8 | 2664.7±54.2 |  |
| m-replay | 2729.9±241.5 | 1258.5±1015.4 | 2855.0±242.2 |  |
| m-expert | 3665.2±74.0 | 3601.0±155.6 | 3889.7±81.6 |  |

#### B.5.2 Winrates

**Table 18: Impact of $f$-divergence on winrates for ComaDICE and baselines in SMACv1.**
| Instances | $f_{\chi^{2}}(x)$ | $f_{\text{KL}}(x)$ | $f_{\text{Soft-}\chi^{2}}(x)$ |  |
| --- | --- | --- | --- | --- |
| 2c_vs_64zg | poor | 0.0±0.0 | 0.0±0.0 | 0.6±1.3 |
| medium | 13.1±4.6 | 10.6±3.8 | 8.8±7.0 |  |
| good | 55.6±3.1 | 54.4±1.5 | 55.0±1.5 |  |
| 5m_vs_6m | poor | 3.8±3.1 | 3.8±3.6 | 4.4±4.2 |
| medium | 6.2±2.8 | 5.0±3.8 | 7.5±2.5 |  |
| good | 8.8±3.6 | 6.9±3.1 | 8.1±3.2 |  |
| 6h_vs_8z | poor | 0.0±0.0 | 0.0±0.0 | 1.9±3.8 |
| medium | 5.0±2.5 | 5.0±3.8 | 3.1±2.0 |  |
| good | 9.4±4.4 | 9.4±2.0 | 11.2±5.4 |  |
| corridor | poor | 1.2±1.5 | 1.2±1.5 | 0.6±1.3 |
| medium | 31.2±6.2 | 28.1±5.9 | 27.3±3.4 |  |
| good | 49.4±5.4 | 48.1±1.5 | 48.8±2.5 |  |

**Table 19: Impact of $f$-divergence on winrates for ComaDICE and baselines in SMACv2.**
| Instances | $f_{\chi^{2}}(x)$ | $f_{\text{KL}}(x)$ | $f_{\text{Soft-}\chi^{2}}(x)$ |  |
| --- | --- | --- | --- | --- |
| Protoss | 5_vs_5 | 52.5±4.1 | 46.2±7.2 | 46.2±6.1 |
| 10_vs_10 | 48.1±7.6 | 55.0±9.8 | 50.6±8.7 |  |
| 10_vs_11 | 22.5±8.7 | 20.6±6.1 | 20.0±4.2 |  |
| 20_vs_20 | 38.1±2.3 | 41.2±7.8 | 47.5±7.8 |  |
| 20_vs_23 | 16.9±4.2 | 15.0±3.6 | 13.8±5.8 |  |
| Terran | 5_vs_5 | 41.2±7.2 | 38.8±10.6 | 30.6±8.2 |
| 10_vs_10 | 30.6±4.1 | 36.2±10.8 | 32.5±5.8 |  |
| 10_vs_11 | 15.6±11.5 | 15.0±7.5 | 19.4±5.4 |  |
| 20_vs_20 | 33.8±6.4 | 28.7±11.8 | 29.4±3.8 |  |
| 20_vs_23 | 5.6±4.1 | 8.1±4.2 | 9.4±5.2 |  |
| Zerg | 5_vs_5 | 29.4±9.0 | 33.1±13.3 | 31.2±7.7 |
| 10_vs_10 | 31.2±7.7 | 26.2±5.1 | 33.8±11.8 |  |
| 10_vs_11 | 11.2±1.5 | 16.2±7.2 | 19.4±3.6 |  |
| 20_vs_20 | 7.5±3.2 | 11.2±7.0 | 9.4±6.2 |  |
| 20_vs_23 | 10.6±3.2 | 10.0±2.3 | 11.2±4.2 |  |

### B.6 Ablation Study: Different Types of Mixer Network

In this section, we explore the impact of using different types of mixer networks within the ComaDICE algorithm. We introduce two settings for the mixer network within the ComaDICE algorithm: 1-layer and 2-layer settings. The mixer network plays a crucial role in aggregating local value functions into a global value function, which is essential for effective policy optimization in multi-agent reinforcement learning (MARL) settings. By examining various mixer network architectures, we aim to understand how these configurations affect the performance and stability of the ComaDICE algorithm.
The comparisons are presented in Tables 20-24, showing both average returns and win rates. The results clearly demonstrate that the 1-layer configuration performs better, providing more stable training outcomes across nearly all tasks. This contradicts the findings in many prior online MARL studies  (Rashid et al., ; Son et al., ; Wang et al., ), which may be due to over-fitting issues in offline learning.

#### B.6.1 Returns

**Table 20: Average returns for ComaDICE and baselines on SMACv1 with different mixer settings.**
| Instances | ComaDICE (ours) |  |  |
| --- | --- | --- | --- |
| 1-layer | 2-layer |  |  |
| 2c_vs_64zg | poor | 12.1±0.5 | 11.5±0.9 |
| medium | 16.3±0.7 | 11.2±0.8 |  |
| good | 20.3±0.1 | 9.0±2.2 |  |
| 5m_vs_6m | poor | 8.1±0.5 | 3.8±1.1 |
| medium | 8.7±0.4 | 0.8±0.3 |  |
| good | 8.7±0.5 | 7.7±0.1 |  |
| 6h_vs_8z | poor | 11.4±0.6 | 10.3±0.3 |
| medium | 12.8±0.2 | 9.1±0.6 |  |
| good | 13.1±0.5 | 8.3±0.5 |  |
| corridor | poor | 6.4±0.5 | 1.5±0.7 |
| medium | 12.9±0.6 | 3.9±1.7 |  |
| good | 18.0±0.1 | 2.6±2.3 |  |

**Table 21: Average returns for ComaDICE and baselines on SMACv2 with different mixer settings.**
| Instances | ComaDICE (ours) |  |  |
| --- | --- | --- | --- |
| 1-layer | 2-layer |  |  |
| Protoss | 5_vs_5 | 14.4±1.1 | 10.5±1.4 |
| 10_vs_10 | 14.6±1.8 | 11.2±1.6 |  |
| 10_vs_11 | 13.2±0.9 | 9.5±0.4 |  |
| 20_vs_20 | 14.8±1.0 | 9.5±0.9 |  |
| 20_vs_23 | 13.3±0.9 | 7.1±2.2 |  |
| Terran | 5_vs_5 | 10.7±1.5 | 8.3±0.8 |
| 10_vs_10 | 11.8±0.9 | 8.8±1.1 |  |
| 10_vs_11 | 9.4±0.9 | 6.4±1.2 |  |
| 20_vs_20 | 11.8±0.5 | 7.8±0.9 |  |
| 20_vs_23 | 8.2±0.7 | 6.6±0.9 |  |
| Zerg | 5_vs_5 | 10.7±2.0 | 7.8±1.1 |
| 10_vs_10 | 11.5±1.0 | 9.7±0.6 |  |
| 10_vs_11 | 11.0±0.9 | 7.9±0.7 |  |
| 20_vs_20 | 9.4±1.2 | 7.8±0.6 |  |
| 20_vs_23 | 10.5±0.8 | 8.0±0.5 |  |

**Table 22: Average returns for ComaDICE and baselines on MaMujoco with different mixer settings.**
| Instances | ComaDICE (ours) |  |  |
| --- | --- | --- | --- |
| 1-layer | 2-layer |  |  |
| Hopper | expert | 2827.7±62.9 | 483.7±349.7 |
| medium | 822.6±66.2 | 648.4±245.9 |  |
| m-replay | 906.3±242.1 | 441.9±260.8 |  |
| m-expert | 1362.4±522.9 | 402.3±288.2 |  |
| Ant | expert | 2056.9±5.9 | 1583.0±160.4 |
| medium | 1425.0±2.9 | 1198.9±53.9 |  |
| m-replay | 1122.9±61.0 | 1041.8±38.4 |  |
| m-expert | 1813.9±68.4 | 1426.6±171.4 |  |
| Half<br>Cheetah | expert | 4082.9±45.7 | 2159.4±658.0 |
| medium | 2664.7±54.2 | 2026.7±244.3 |  |
| m-replay | 2855.0±242.2 | 1299.2±196.1 |  |
| m-expert | 3889.7±81.6 | 1336.3±381.9 |  |

#### B.6.2 Winrates

**Table 23: Average winrates for ComaDICE and baselines on SMACv1 with different mixer settings.**
| Instances | ComaDICE (ours) |  |  |
| --- | --- | --- | --- |
| 1-layer | 2-layer |  |  |
| 2c_vs_64zg | poor | 0.6±1.3 | 0.0±0.0 |
| medium | 8.8±7.0 | 3.8±3.6 |  |
| good | 55.0±1.5 | 19.4±5.0 |  |
| 5m_vs_6m | poor | 4.4±4.2 | 3.1±0.0 |
| medium | 7.5±2.5 | 1.2±1.5 |  |
| good | 8.1±3.2 | 3.1±0.0 |  |
| 6h_vs_8z | poor | 1.9±3.8 | 0.0±0.0 |
| medium | 3.1±2.0 | 0.0±0.0 |  |
| good | 11.2±5.4 | 1.9±2.5 |  |
| corridor | poor | 0.6±1.3 | 0.0±0.0 |
| medium | 27.3±3.4 | 11.2±2.5 |  |
| good | 48.8±2.5 | 23.1±8.1 |  |

**Table 24: Average winrates for ComaDICE and baselines on SMACv2 with different mixer settings.**
| Instances | ComaDICE (ours) |  |  |
| --- | --- | --- | --- |
| 1-layer | 2-layer |  |  |
| Protoss | 5_vs_5 | 46.2±6.1 | 31.9±3.6 |
| 10_vs_10 | 50.6±8.7 | 32.5±5.8 |  |
| 10_vs_11 | 20.0±4.2 | 10.6±7.3 |  |
| 20_vs_20 | 47.5±7.8 | 21.9±4.0 |  |
| 20_vs_23 | 13.8±5.8 | 6.9±5.4 |  |
| Terran | 5_vs_5 | 30.6±8.2 | 25.6±4.6 |
| 10_vs_10 | 32.5±5.8 | 28.1±3.4 |  |
| 10_vs_11 | 19.4±5.4 | 12.5±4.0 |  |
| 20_vs_20 | 29.4±3.8 | 11.2±3.2 |  |
| 20_vs_23 | 9.4±5.2 | 3.1±2.0 |  |
| Zerg | 5_vs_5 | 31.2±7.7 | 20.6±4.7 |
| 10_vs_10 | 33.8±11.8 | 21.2±7.2 |  |
| 10_vs_11 | 19.4±3.6 | 13.1±4.1 |  |
| 20_vs_20 | 9.4±6.2 | 5.6±1.3 |  |
| 20_vs_23 | 11.2±4.2 | 3.1±3.4 |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: The ablation study shows that performance decreases when using a more complex network. Why is that the case? If more data were available or the model was improved, could the results potentially differ?
soundness: 3
strengths: - Effective Distribution Correction in Multi-Agent Settings: ComaDICE improves upon traditional DICE by extending stationary distribution correction to multi-agent environments. Through f-divergence-based alignment between behavior and target policies, it effectively handles the complex distributional shifts unique to multi-agent interactions, enhancing policy reliability and performance.
- Theoretical Foundation: A thorough mathematical analysis provides convergence and stability proofs through f-divergence correction, reinforcing the algorithm’s reliability in multi-agent scenarios.
- Stable Value Decomposition: ComaDICE decomposes global values into convex local objectives, enhancing training stability and aligning local agent optimization with the global objective to address coordination and stability challenges specific to multi-agent environments.
summary: This paper proposes an algorithm introducing stationary distribution correction to address the distributional shift problem in offline cooperative multi-agent reinforcement learning (MARL). In multi-agent environments, this issue is intensified due to the large joint state-action space and the interdependencies among agents. To tackle this, ComaDICE minimizes the f-divergence between the stationary distributions of the learning and behavior policies. Additionally, by leveraging the Centralized Training with Decentralized Execution (CTDE) framework, it decomposes the global value functions into local values for each agent, ensuring that the optimization of each agent’s local policy is consistent with the global learning objective.
weaknesses: - ComaDICE’s theoretical analysis relies on non-negative weights and convex activations in the mixing network for stability. While this aids convergence, it may limit the model’s ability to capture complex inter-agent dynamics. In the practical algorithm, this limitation is intensified by the use of a single-layer linear mixing network, which further restricts representational capacity in highly interactive environments.
- High Computational Cost: ComaDICE incurs a significant computational cost due to the precise f-divergence-based adjustments needed for each agent’s local policy in stationary distribution correction. This can lead to reduced efficiency, especially in environments with a large number of agents.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. What is the main purpose for the value decomposition and learning individual Q-functions? 

1. Related to the first question, what is the purpose of Theorem 4.2, if the original loss is also concave? If we use the mixing functions, does $ \mathcal{\tilde L}$ still find the global optimum?

1. Can ComaDICE solve the XOR game and Bridge mentioned in AlberDICE? 

1. Related to “Strengths”, can solving ComaDICE learn the global optimum in the underlying MDP even with the mixing network?


1. Is the Individual Global Max (IGM) assumption required in order to introduce the mixing network? In other words, does ComaDICE assume that the underlying optimal Q function assume IGM?

1. What is the main difference in the final algorithm with OMIGA [3] ?

1. Why are the notations mixed between using joint observations $\mathbf{o}$ and $s$? Is the setting a Dec-POMDP or a specialized setting where the joint observations constitute the state?

1. Please write in a different color during the rebuttal which part is the novel part. In particular, which part of the proofs are a novel contribution and which part is coming from OptiDICE [2]

1. Please address both my comments in the Strengths and Weaknesses section in detail.


### References
[1] AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation (Matsunaga et, al. NeurIPS 2023)

[2] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation (Lee et, al. ICML 2021)

[3] Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization (Wang et, al. NeurIPS 2023)

----------------------------------------------------------------------------------------------------------------------
Post-Rebuttal
----------------------------------------------------------------------------------------------------------------------

During the rebuttal, the authors worked hard to address my concerns, namely (1) additional baselines (AlberDICE and OptiDICE) on all 3 environments, (2) additional toy example (XOR Game), (3) a re-interpretation of the method as learning the optimal policy implicitly in factorized policy space and (4) detailed analysis on how ComaDICE works based on the XOR Game which is a toy domain. 

The reason why my initial score was low was because a lot of these points were not explicit in the original draft, and it was unclear what the contribution was. After the rebuttal and some discussion, these points were made clear and the authors demonstrated that (1) using a factorized approach to decomposing stationary distributions is implicitly searching for optimal factorized policies (2) ComaDICE is able to approximate a global optimum as opposed to Nash policies (AlberDICE), (3) ComaDICE outperforms all baselines in all environments including AlberDICE and OptiDICE, which are most closely related, (4) the algorithm is more scalable and simple compared to AlberDICE.

As it currently stands, these contributions of the paper are scattered and it would take some effort by the reader to appreciate the paper, without some background in both DICE-based approaches in MARL. I would recommend a re-write of the storyline of the paper, highlighting the fact that ComaDICE pushes the cooperative Offline MARL field forward beyond IGM as well as AlberDICE.  

As a result, I've increased my score as follows:

Soundness: 2-->3

Presentation: 1-->2

Contribution: 1--> 3

Overall Score: 3: reject, not good enough --> 6: marginally above the acceptance threshold

I can't quite give an 8 (Strong Accept) due to the writing, but I would have given 7 if that option was available.
soundness: 3
strengths: It is known that every MDP has a deterministic optimal policy, which, if extended to the multi-agent setting, can be factorized into independent policies. A very optimistic reading of the paper would interpret that the decomposition procedure is actually learning optimal value functions over decentralized policies. If this interpretation is correct, the ComaDICE presents a very scalable and principled offline MARL algorithm for learning decentralized value functions and policies without any restrictive IGM assumption as in previous work such as ICQ. However, it is worth noting that this is more of a statement of the potential of the paper rather than a strength of its current version, as these points are not specifically addressed in the current draft.

Furthermore, ComaDICE can be more scalable in comparison to AlberDICE [1] which requires alternating optimization.
summary: The paper proposes ComaDICE, a stationary distribution correction estimation approach to addressing OOD states and actions in Offline MARL. The derivation starts with the LP formulation of RL in the joint state-action space, which results in a concave objective for learning state-value functions similar to OptiDICE [2]. The objective also uses value decomposition with a monotonic mixing network for the advantage function similar to OMIGA [3]. ComaDICE is evaluated on SMACv2 and MaMuJoCo and shows comparable performance with baselines such as BC and OMIGA.
weaknesses: ### Problem Statement
It is not clear what the main problem is that ComaDICE is solving. It is briefly mentioned in the introduction that OOD states and actions are a problem in offline MARL. However, this was addressed in detail by other work such as CFCQL/OMIGA/AlberDICE mentioned in the Related Work (especially AlberDICE [1] which is the most similar). For instance, AlberDICE considers some coordination problems (XOR/Bridge/etc.) where OOD joint actions may be common, these is no consideration of these settings as well as comparison to AlberDICE both algorithmically and empirically. Thus, it is not entirely clear from the current draft of the paper what the main problem is that ComaDICE is solving, and if it is indeed OOD actions, which part of the algorithm in particular is alleviating this.

### Novelty
The derivation is based on OptiDICE [2] but this is not explicitly mentioned, which is misleading. 
For instance, Proposition 4.1 seems equivalent to Proposition 1 of OptiDICE. Furthermore, an extension of OptiDICE to solve Offline MARL was considered in AlberDICE [1] so it is not clear why ComaDICE should be preferred over AlberDICE. Furthermore, the final algorithm closely resembles OMIGA [3].

### Algorithm
It is unclear what the purpose of value decomposition in learning the Q functions is. It seems the advantage can be computed by the learned state-value function $\nu$ and run WBC (as mentioned in Appendix C of AlberDICE[1]). Also, Line 346 defines $A_\nu^{tot}$ as the sum of reward and state-value functions.

### Lack of Relevant Baselines
AlberDICE and OptiDICE are missing as the main baselines, as well as CFCQL which addresses OOD actions but in a different manner. These are all mentioned in the Related Work section but not compared.

### Writing
The paper is generally not well-written. All of the aforementioned weaknesses of the paper should be addressed in detail and the writing should not raise any of these concerns.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: * The discussion of the DICE method and its goals in the Related Work and Preliminary Knowledge section is not sufficiently in-depth to clearly locate the innovations of this paper with respect to existing work.
* The proof section of DICE skips some key steps,  has a confusingly organized derivation and notation that does not clearly state the purpose and rationale for each step of mathematical transformation. Interpretation between Lagrange functions and offline reinforcement learning problem formulation is in lack.
* The paper fails to explore the common sparse reward scenario in offline reinforcement learning, and the analysis of the quality requirements of behavioral strategies is insufficient, limiting the feasibility of the method in practical applications.
soundness: 3
strengths: * The introduction of the DICE algorithm within the domain of multi-agent reinforcement learning represents a significant contribution characterized by its novelty and innovative approach.
* The experimental settings are comprehensive, demonstrating the potential of the proposed algorithm effectively.
summary: In this paper, ComaDICE algorithm is proposed for offline multi-agent reinforcement learning, extending the DICE method to multi-agent scenarios and using value function decomposition and policy extraction methods, the idea is novel and the experimental results show the potential of the method.
weaknesses: * The contribution emphasizes the value decomposition and claims to prove the equivalence of the local policy product to the globally optimal policy, but the paper is too repetitive and superficial in its exposition of the decomposition method, neither explicitly defining the local subtasks and their corresponding policies, nor the relationship between them, nor providing a complete proof of this equivalence.
* The experimental results for hybrid networks contradict mainstream research findings (single-layer outperforms double-layers), a phenomenon that hints at a possible fundamental flaw in the application of the DICE methodology to MARL, but the paper lacks an in-depth discussion of this.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Q1. Do the authors assume that $s$ can be sufficiently represented from $o$?
Notations for states and observation are confusingly used. For example, in line 234 and 237, $\nu(s)$ and $q(s,a)$ are defined as a collection of functions that requires individual observations,$\nu_i(o_i)$ and $q(o_i,a_i)$ respectively If so, the assumption should be explicitly clarified in the POMDP description.

Q2. Figure 2 lacks information on which task was selected for each benchmark. (e.g. 5_vs_5 or 10_vs_10 in protoss, or expert or medium data quality in Hopper) Could you clarify?
soundness: 3
strengths: This work provides a comprehensive performance evaluation through experiments conducted on an extensive set of benchmarks, including the challenging multi-agent RL benchmark SMAC-v2. The experimental results indicate that the proposed approach shows superior performance relative to the baselines considered in the manuscript.
summary: This work introduces ComaDICE (offline Cooperative MARL with DICE), an approach for offline cooperative multi-agent reinforcement learning (RL) that leverages the DICE method. ComaDICE formulates the offline cooperative multi-agent RL problem as a constrained optimization and employs a DICE-based method to compute a global Lagrangian multiplier, $\nu^{tot}$. Given the large state and action spaces typical in multi-agent RL, practical optimization of $\nu^{tot}$ may not be feasible. To address this challenge, ComaDICE employs value function decomposition to decompose the global Lagrangian multiplier into individual Lagrangian multipliers for each agent.
weaknesses: The primary concern regarding this work is its novelty compared to previous work.

Specifically, AlberDICE by Matsunaga et al. (2023) [A] conveys a similar idea, i.e. adoption of DICE for offline cooperative multi-agent RL. The key difference of ComaDICE from AlberDICE appears to be employing individual Langrangian multipliers $\nu_i$ and the mixing network for value function decomposition: while AlberDICE utilizes a simple yet principled resampling method for obtaining $\nu_i$, ComaDICE employs the value factorization as discussed in Section 4.2, which requires the reliance on the additional mixing network $\mathcal{M}_\theta$ further necessitates additional training.

Although AlberDICE is briefly mentioned in Section 2, the paper does not adequately discuss the theoretical or empirical advantages of adopting the value decomposition instead of the resampling approach. In addition, the AlberDICE paper presents a simple multi-agent task where value function decomposition leads to a substantial performance loss.
 
The absence of such a comparison significantly weakens the perceived contribution of this study, as it fails to establish a clear improvement of differentiation from previous work. In this sense,
(1) A detailed discussion comparing ComaDICE and AlberDICE should be added,
(2) AlberDICE should be included as a baseline in all experiments,
(3) ComaDICE should be tested on the XOR game in the AlberDICE paper  


[A] Matsunaga et al., “AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation.”, NeurIPS 2023.

[Minor comments]
In Sections 6.1 and 6.3, SMACv1 environment is discussed; however, corresponding experimental results for SMACv1 are not included in the main manuscript. It would be more appropriate to relocate these discussions to the appendix and direct readers there for further details, or incorporate the result on SMACv1 to the main manuscript, potentially replacing some of the redundant results on SMACv2 in Table 1 or Figure 1.

Meta-review: The paper studied offline multi-agent RL and proposed an approach based on the DICE framework. The proposed approach uses a stationary distribution shift regularization to combat the distribution shift issue in offline RL. The paper demonstrates that their approach works well empirically.

Decision: Accept (Poster)