ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
Authors: The Viet Bui
Thanh Hong Nguyen
Tien Anh Mai
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement Learning
Multi-Agent Reinforcement Learning
Stationary Distribution Correction Estimation
Primary area: reinforcement learning
TL;DR: This paper introduces ComaDICE, a novel offline cooperative multi-agent reinforcement learning algorithm that uses stationary distribution shift regularization to improve performance in complex environments like MuJoCo and StarCraft II.

Abstract: Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we propose a novel type of regularizer in the space of stationary distributions to address the distributional shift more effectively. Our algorithm, ComaDICE, provides a principled framework for offline cooperative MARL to correct the stationary distribution of the global policy, which is then leveraged to derive local policies for individual agents. Through extensive experiments on the offline multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.

Published as a conference paper at ICLR 2025
COMADICE: OFFLINE COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING WITH STATIONARY DIS- TRIBUTION SHIFT REGULARIZATION
The Viet Bui
Hong Thanh Nguyen
School of Computing and Information Systems Singapore Management University, Singapore theviet. bui 20220 22w.@phcc.............cccccccc.3 smu edu. sg
University of Oregon Eugene, Oregon United States
thanhhng@cs wwteeccs.rrroooo orgeon. edu
Tien Mai
School of Computing and Information Systems Singapore Management University, Singapore atmai@smu edu. sg
ABSTRACT
Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint acte-nction space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce regularizer in the space of stationary distributions distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi- agent value decomposition strategy facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft // benchmarks, we demonstrate that ComaDICE achieves superior performance compared state- of-the-art offline MARL methods across nearly all tasks.
better handle
INTRODUCTION
Over the years, deep RL has achieved remarkable success in various decision-makin tasks (Levine) ct al.] 22166) Silver et al. 2017: Kalashnikov et al. 2018 Haydari & Yılmaz 2020). However, significant limitation of deep RL need for millions of interactions with the environment gather experiences for policy improvement. This process can be both costly and risky, especially in real- world applications like roboties and healthcare. To address this challenge, offline RL has emerged, enabling policy learning based solely pre-collected demonstrations (Levine et al. 2020). Despite this advancement, offline RI. faces a critical issue: the distribution shift between the offline dataset and the learned policy (Kumar et al. 22019). This distribution shift complicates value estimation for unseen states and actions during policy evaluation, resulting in extrapolation errors where -fiitirrbiutto (OOD) sate-action pairs are assigned unrealistic values (Fujimoto al. 2018).
To tackle OOD actions, many existing works impose action-level constraints, either implicitly by regulating the learned value functions explicitly through distance divergence penalties (Fujimoto ct al. 2019; Kumar et al., 2019 Wu et al. (2019) Peng et al., 2019 |Fjiimoto & Gul 2021 Xu et al.
2021
Only few recent studies have addressed both OOD actions and states using state-action-level

---

Published as a conference paper at ICLR 2025
behavior constraints (Li et al. 2022 Zhang et al. 2022 Lee et al. 2021 2022
Mao et al. 2024 In
particular, there is an important line of work on DIstribution Correction Estimation (DICE) (Nachum
& Dai 2020) that constrains the distance in terms of the joint state-action occupancy measure between the learning policy and the offline policy. These DICE-based methods have demonstrated impressive
performance results the D4RL benchmarks (Lee al. 2021
2022
Mao al.. 2024).
is important to note that that all the aforementioned offline RL approaches primarily focus on the single-agent setting. While multi-agent setting is prevalent in many real-world sequential decision- making tasks, offline MARL remains a relatively under-explored area. The multi-agent setting poses significantly greater challenges due to the large joint state-action space, which expands exponentially with the number of agents, as well as the cnterdeependsncess among the local policies of different agents. As a result, the offline data distribution can become quite sparse in these high-dimensional joint action spaces, leading to an increased number of OOD state-action pairs and exacerbating extrapolation errors. A few recent studies have sought to address the negative effects of sparse data distribution in offline MARL by adapting the well-known centralized training decentralized execution (CTDE) paradign from online MARL (Olichoek et al. 2008; Kraemer & Banerjee 2016), enabling data-related regularization at the individual agent level. Notably, some of these works (Pan et al., 2022; Shao et al. 20244 Wang et al. (002b) extend popular offline single-agent RL algorithms, such CQL (Kumar et 2020) and SQL/EQL (Xu al. 2023), within the CTDE framework.
In our work, we focus on addressing the aforementioned challenges in offline cooperative MARI. In particular, we follow the DICE approach address both OOD states and actions, motivated by remarkable performance of recent DICE-based methods in offline single-agent RL. Similar to previous works in offline MARL, we adopt the CTDE framework to handle exponential joint state-action spaces in the multi-agent setting. We remark that extending the DICE approach under this CTDE framework is not straightforward given the complex objective of DICE that involves the f-divergence in stationary distribution between the learning joint policy and the behavior policy. Therefore, the value decomposition in CTDE needs to be carefully designed to ensure the consistency in optimality between the global and local policies. In particular, we provide the following main contributions:
We propose ComaDICE, new offline MARL algorithm that integrates DICE with a carefully designed value decomposition strategy. In ComaDICE, under the CTDE framework, we decompose both the global value function fot and the global advantage functions Atot rather than using Q-functions in previous MARL works. This unique factorization approach allows us to theoretically demonstrate that the global learning objective in DICE is convex in local values, provided that the mixing network used in the value decomposition employs non- non-negative weights and convex activation functions. This significant finding ensures that our decomposition strategy promotes an efficient and stable training process.
Building on our decomposition strategy, we demonstrate that finding an optimal global policy can be divided into multiple sub-problems, each aims to identify a local optimal policy for an individual agent. We provide theoretical proof that the global optimal policy is, in fact, equivalent to the product of the local policies derived from these sub-problems.
Finally, we conduct extensive experiments to evaluate the performance of our algorithm, ComaDICE, in complex MARL environments, including: multi-agent StarCraft II (i.e., al., 2019). SMACv2 (Ellis et al., 2 and multi-agent Mu- joco (de Witt et al. 2020) benchmarks. Our empirical results show that our COmaDICE outpertonne outperforms several strong baselines in all these benchmarks.
SMACv1 (Samvelyan
2 RELATED WORK
Offliie Reinforcement Learning (offline RL). Offline RL focuses on learning policies from pre-collected datasets without any further interactions with the environment (Levine et al. 2020 Prudencio et al. 2023). A significant challenge in offline RL is the issue of distribution shift, where unseen actions and states may arise during training and execution, leading to inaccurate policy evaluations and suboptimal outcomes. Consequently, there is a substantial body of literature addressing this challenge through various approaches (Prudencio et al. 2023). In particular, some studies impose explicit or implicit policy constraints to ensure that the learned policy remains close to the behavioral policy (Fujimoto et al. 2019 Kumar et al. 2019 Wu et al. 2019; Kostrikov et al.
202T Peng al. 2019
Nair et al. 2020
Fujimoto & Gu 202 Xu et al.
Cheng al. 2024
2

---

Published as a conference paper at ICLR 2025
al. 2023
Others incorporate regularization terms into the learning objectives to mitigate the value
overestimation on OOD actions (Kumar et al. 2020; Kostrikov et al. 2021; Xu et al. 2022c] Niu et al.
2022, Xu et al. 2023 Wang al.) [2022b). Uncertainty-bust based offline RL methods seek to balance conservative approaches with naive off-policy RL techniques, relying on estimates of model, value, or policy uncertainty (Agarwal et al., (020 An et al. 2021 Bai al., (022). Offline model-based algorithms focus on conservatively estimating the transition dynamics and reward functions based on the pre-collected datasets (Kidambi al., 2020 Yu et al. 2020; Matsushima et al., 2020 Yu et al.) 202I). Some other methods impose action- level re regularization through imitation learning techniques (Xu al. |022b] Chen et al., 2020 Zhang et al. 2023] Zheng et al., 2024 Brandfonbrener et al., 2022a). Finally, while majority of previous works target OOD actions only, there are a few recent works attempt to address both OOD states and actions (Li et al. 2022 ZZhan et al.) 2022 Lee et al.
2021 Xu et al.]
2021 2022 Siikchi et al. 2023 Mao et al. 2024). Our work on offline MARL. follow the DICE- based approach, as motivated by compelling performance of DICE- based algorithms in
single-agent settings (Lee al. 202 2022 Sikehi et al. 2023 Mao et al. 2024
Offiine Multi-agent Reinforcement Learning (offline MARL). While there is a substantial body of literature on offline single-agent RL., research offline MARL remains limited. Offline MARL faces challenges from both distribution chartt--rraattttticcccccc of offline settings- and the exponentially large joint action space typical of multi-agent environments. Recent studies have begun to merge advanced methodologies from both offine RL and MARL to address these challenges (Yang et al.) 2021; Pan et al., 2022; Shao al. 2024 Wang et al. 2022b) Specifically, these works employ local policy regularization within the centralized training with decentralized execution (CTDE) framework to mitigate distribution shift. The CTDE paradigm, well-established in online MARL, facilitates more efficient and stable learning while allowing agents to operate in a decentralized manner (Oliehoek) 2008 Kraemer& & Banerjee| 2016). For instance, (Yang et al. (202T) utilize importance sampling al. to manage local policy learning on OOD samples. Both works by Pan et al. (2022) and [Shao] et al. (2024) are built upon CQL (Kumar et al. 2020p, a prominent offline RL algorithm for single- agent scenarios. Matsunaga et al. (2023) developed AlberDICE, leveraging the Nash equilibrium solution concept from game theory to iteratively update the best responses of individual agents. Both AlberDICE and our method, ComaDICE, adopt the DICE framework to address the out-of diitriiiittiiiooo (OOD) issue. However, while AlberDICE proposes learning individual Lagrange multipliers (or value functions) to obtain occupancy ratios, our ComaDICE. algorithm learns a global value function by mixing local functions, adhering to the well-established CTDE principle. This design enables ComaDICE to better capture inter-agent relationships and improve credit assignment across local agents. Finally, OMIGA (Wang et al.] 2022b) establishes the equivalence between global and local value regularization within a policy constraint framework, making the current state-of of-tte------rr algorithm in offline MARL. The key difference between ComaDICE and OMIGA lies in their respective approaches: OMIGA focuses on learning a global Q-function, whereas our algorithm (and other methods in the DICE family) operates in the occupancy space, aiming to learn the ratio between the occupancy of the learning policy and the behavior policy.
iistance,[Yang
Beyond this main line of research, some studies formulate offline MARL sequence modeling problem, employing supervised learning techniques to tackle the issue (Meng el al. 2023 Tseng
et al. 2022). while others adhere to decentralized approaches (Jiang & Lu 2023).
3 PRELIMINARIES
Our work focuses on cooperative multi-agent RL, which can be modeled as a multi-agent Partially Ob- servable Markov Decision Process (POMDP), defined by the tuple M = (S, A, r, Z, O, N, ..,)). Here, n is number of agents, NV = {1.....7 n. is the set of agents, s E S represents the true state of the multi-agent environment, and A Ieey A, is the set of joint actions, where A, is the set of individual actions available to agent i E N. At each time step, each agent í E (1,2.....n) n} selects an action a; E A,, forming a joint action a = (a), a2.... an)) E A. The transition dy- namics P(s'(s, a) S x A S - [0, I] describe the probability of transitioning to the next state s' when agents take an action a from the current state s. The discount factor y E [0, 1) represents the weight given to future rewards. In partially observable environment, each agent receives local observation s; E O based on the observation function (s) S - O, and we denote the joint observation as 0 (01, on)) In cooperative MARL, all agents share global reward
n)-
3

---

Published as a conference paper at ICLR 2025
function r(s, a) 8 A - RR. The goal of all agents is to learn a joint policy Tot { ......... Tr, } that collectively maximize the expected discounted returns ((oa))= 1Ex= "r((s,, az)]- In the offline MARL setting, pre-collected dataset D is obtained by sampling from behavior policy Hhot = (M1,-- Pn J. and the policy learning is conducted soly based on DD with no interactions with the environment. We also define the occupancy measure (or stationary distribution) follows:
r (s, a) = =(1 (1-7) Σ
P(s: = =
1=0
which represents distribution visiting the pair (observation, action) (Sc, a1) when following the joint policy tott where So Po: a, tooe (-o((((s)) (sx) and 8141 P(-Is,, at).
4 COMADICE: OFFLINE COOPERATIVE MULTI-AGENT RL WITH STATIONARY DISTRIBUTION CORRECTION ESTIMATION
We consider an offline cooperative MARL problem where the goal is to optimize the expected discounted joint reward. In this work, we focus the DICE objective function Nachum
JunctionNNachu & Dai (2020) Lee et al. (2021), which incorporates a stationary distribution regularizer to capture the divergence between the occupancy measures of the learning policy, tot. and the behavior policy, Pooor formulated as follows:
maxx (ss [r(s. a)] -a (Di (M)))
where D' (p || pftoot = B(s.a))pttor proe is the f-divergence between the stationary dis- tribution tht the learning policy and p of the behavior policy. In this work, we consider f(-) to be strictly convex and differentiable. The parameter a controls the trade-off between maximizing the reward and penalizing deviation from the offline dataset's distribution (i.e., penalizing distributional shift). When = 0, the problem becomes the standard offline MARL, where the objective is to find a joint policy that maximizes the expected joint reward. On the other hand, when a >> 1, the problem shifts towards imitation learning, aiming to closely mimic the behavioral policy.
prece
This DiCE-based approach offers the advantage of better capturing the system dynamics inherent in the offline data. Such stationary distributions, not and pltio,, however, are not directly available. We will discuss how to estimate them in the next subsection.
4.1 CONSTRAINED OPTIMIZATION IN THE STATIONARY DISTRIBUTION SPACE
We first formulate the learning problem in Eq. D as constrained optimization on the space of preat:
maxprcot B(s.a)-ptton [r(s, a)] oD (p || p
s.t. Σn (s,
prods (s, (1 ())osssss))))))))))))))))))>))) Σ Ex a'sl Vs E S.
(3)
When f is convex, becomes a convex optimization problem, as it involves maximizing a concave objective function subject to linear constraints. We now consider the Lagrange dual of 23:
[
(Ntto((s (s, phioe (s.
Ll p Fsui)
E (sa))upt (r(s,a)) aI (s))s))ppe
Σ uo (s) E
p Troe (s, a') ()-o) En
p Toe (s' assla'' (4)
where y (s) is a Lagrange multiplier. Since 2 is convex optimization problem, it is equivalent to the following minimax problem over the spaces of not and T min max {L(voo Tent
Furthermore, we observe that L( x(oe)) is linear in yoo and concave in tutt so the minimax problem has a saddle point, implying: min maxp [L(yto (eoe)) max mitlyto [L(wrot F }} In a manner analogous to the single-agent case (Lee et al. 2021 by defining wrot(s a) *oor (5.a)
priot (s.a) the Lagrange dual function can be simplified into the more compact form (with detailed derivations are in the appendix):
C(vtot zo0) (1 7) ( (s) + ((s.a))-Pror aa (w (s. a)) + " (s,a) (s. (s. a))
4

---

Published as a conference paper at ICLR 2025
where Abet is an "advantage function" defined based psfo as:
A (s, a) = "5n a) (s))
with (s. a) (()(((()))))))))) 7 (sa)) (s)] It important to note that ('o and gead (s. a) can be interpreted value function and a Q function, respectively, arising from the decomposition of the stationary distribution regularizer. We can now write the learning problem as follows:
ming max
{C(v"ot w (.
(6)
can be observed that C(vtot woot) is linear in yo and concave in whtt which ensures well- behaved properties in both the y²at_ and w -saces. Following the derivations in Lee et al. (2021) a key feature of the above minimax problem is that the inner maximization problem has a closed- form solution, which greatly simplifies the minimax problem, making it no longer adversarial. We formalize this result as follows:
Proposition 4.1. The minimax problem in Eq. 6 equivalent to min
{E(vtat),, where
E(y - -1)-- tot (s)] + E aaamppeee aas*
Alot
Here, f is convex conjugate of f. i.e., f (v) supizolty- f(t), Moreover vft is paaaameteiized by O, the first-order derivative of ((v)ot) w.r.l. O given as follows:
VoL(vtot) (1-))1Ey--p) Vortot(s) ] Eisa))pproo VoA (s, (s.)tttttsss)))
where who (s) a) = max (0, f f' (A (s, a) a))a), with (-) the inverse function of the first-
order derivative of
Proposition 4.1 above is a direct extension of the formulations in Lee et al. (021) developed for the single-agent setting, differing only in the inclusion of the closed-form expression for the first-order derivative of the objective function, C(vtot).
4.2 VALUE FACTORIZATION
Directly optimizing min [C(ptet , (ottt) in multi-agent settings is generally impractical due to the large state and action spaces. Therefore, we follow the idea of value decomposition in the well- known CTDE framework in cooperative MARL to address this computational challenge. However, it not straightforward to extend the DICE approach within this CTDE framework due to the complex objective of DICE, which involves the f-divergence between the learned joint policy and the behavior policy in stationary distributions. Thus, crucial to carefully design the value decomposition in CTDE to ensure optimality consistency between the global and local policies.
Specifically, we adopt a factorization approach that decomposes the value function poo (s) (or global Lagrange multipliers) into local values using mixing network architectures. Let represent collection of local "value functions" and let A a) = 1...., n represent collection of local advantage functions. The local ad- vantage functions are computed as A,(s;,
v(s) = (..) {Ar a.),
qi(8;, a;) v;(s;) for all E N, where 1....,91 n) is vector of local Q functions. To facilitate centralized learning, we create a mixing network, Mo, where are the learnable weights, that aggregates the local values form the global value and advantage functions follows:
(ss a) = (si a;), i
tot (s, a) = Mo(u(s)), Ato a) Malq(s, a) v(s)],
where each network takes the vectors (s) or (s,s, a) as inputs and outputs for and Atet, respectively. Under this architecture, the learning objective becomes:
T(v, 0) = (1 71-7)) Es [M [s)] + (ss)) Tas-
-
with the observation that A, As a) can be expressed as linear function of v. There are different ways to construct the mixing network Ma; previous work often employs a single linear combination (1 (I-layer network)
two-layer network with convex activations such ReLU, ELU, or Maxout. In the following, we show a general result stating that the learning objective function is convex in v, provided that the mixing network is constructed with nonnegative weights and convex activations.
5

---

Published as a conference paper at ICLR 2025
Theorem 4.2. If the mixing network Mal- is constructed with non-negative weights and convex activations, then C(v, 0) is convex in v.
Mixing networks with non-negative weights and concave activations (e.g.. ELU or RcLU) have been extensively used in MARL, forming the foundation of several notable state-of tfetthhhann art algorithms such as QMIX (Rashid et al. 2020), QTRAN (Son et al. (2019), and MFIQ (Bui et al. 2024). In particular, it has been demonstrated that mixing networks with either negative weights or non-concave activations
result in significantly degraded performance (Bui et al. 2024). Theorem 4.2 shows that C(v, 0) is convex in y when using any multi-layer feed-forward mixing networks with non-negative weights and convex activation functions. This finding is highly general and non-trivial, given the nonlinearity and complexity of both the function (in terms of v) and the mixing networks. Previous work has often focused on single-layer (Wang et al.) 2022b) or two-layer mixing structures (Rashid et al. 2020: Bui ct al. 2024). emphasizing that such two-layer networks can approximate any monotonic function arbitrarily closely as network width approaches infinity (Dugas et al., 2009). In our experiments, we test two configurations for the mixing network: a linear combination (or 1-layer) and a 2-layer feed-forward network. While 2-layer mixing structures have shown strong performance in online
MARL (Rashid et al. 2020; Sonn et al. 2019 Wang et al.] (020), we observe in our offline settings that the linear combination approach provides more stable results.
4.3 POLICY EXTRACTION
Let y" be an optimal solution to the training problem with mixing networks, i.e.,
min Z(v,0). 0).
v,0
We now need to extract local and joint policy from this solution. Based on Prop. 4.1 given v*, we can compute this occupancy ratio follows: wot (s. a) max (0, (' Mo (A, (s.a)
The global policy can then be obtained as follows: Ti Evea wotor (s.a") pussor (s.a') This computation, however, is not practical since plror is generally not available and might not be accurately estimated
a
w w (P.o))
in the offline setting. A more practical way to estimate the global policy, Toor as the result of solving the following weighted behavioral cloning (BC):
max E Went ETTee (s.a)>p"iuu
[log too (a
max E(sa)) tote (s. a) log too (a[s)). Eloot
(8)
where Hhst represents the feasible set of global policies. Here we assume that Uo contains decom- posable global policies, i.e., tot [Toor 3x, Vi E such that Too (als) Ihe Ty (o(o.(s)) In other words, ,f consists of global policies that can be expressed as a product of local policies. This decomposability is highly useful for decentralized learning and has been widely adopted in MARL (Wang et al. 2022b Bui et al. 2024 Zhang et al. (021).
While the above weighted BC appears practical, (s, a) be sampled from the offline dataset generated by pxtet, and since wy (s, a) is available from solving 7 it does not directly yield local policies, which are essential for decentralized execution. To address this, we propose solving the following weighted BC for each local agent i E N:
max;, E(s.a)DD [ (s. a) log (a:ls)))
This local WBC approach has several attractive properties. First, to (s. a) appears explicitly in the local policy optimization and is computed from global observations and actions. This enables local policies to be optimized with global information, ensuring consistency with the credit assignment in the multi-agent system. Furthermore, shown in Proposition 4.3 below, the optimization of local policies through local WBC is highly consistent with the global weighted BC in g
Proposition 4.3. Let πद be the optimal solution to the local weighted BC Then Ti (als)
ThEN m (a,ls;) ) is also optimal for the global weighted BC in
Here we note that consistency between global and local policies is a critical aspect of centralized training with CTDE. Previous MARL approaches typically achieve this by factoring Q or V functions into local functions and training local policies based on these local functions (Rashid et al.) 2020 Wang et al. 2020 Bui al. 2024). However, in our case, there are key differences that prevent us

---

Published as a conference paper at ICLR 2025
from employing such local values to derive local policies. Specifically, we factorize the Lagrange multipliers 2ato to train the stationary distribution ratio wint, Although local w values can be extracted from local F4, these local values do not represent local stationary distribution ratio and therefore cannot be used to recover local policies.
5 PRACTICAL ALGORITHM
Let D represent the offline dataset, consisting of sequences of local observations and actions gathered from global behavior policy Toot- To train the value functiin v, construct a value network pa(s;; thu) for each local agent i, along with a network for each local Q-function q: (8; d;; thy), where t and V/ learnable parameters for the local value and Q-functions. We note that the introduction and learning of the Q-functions are intended to facilitate the decomposition of the advantage function, Atot, In our multi-agent setting, the absence of local rewards makes it difficult directly compute local advantage functions, To overcome this challenge, we learn local Q-functions, which are then used derive the local advantage functions. Additionally, as explained below, a MSE is optimized to ensure that the global Q-function and state-value function align properly with the global rewards.
Now, each local advantage function is then calculated as follows: The global value function and advantage function are subsequently aggregated using two mixing networks with shared set of learaalle parameters 8:
y rot (s) = M (s; 1.),, A a) (alg(s, a: 1.,) v(s; v.))
where MM[+-] represents a linear combination of its inputs with non-negative weights, such that My (s; ) (s; 2 TR)TWW W7 + by. where W and bé are weights of the mixing network]] ! It is important to note that W and bg are generated by hyper-networks that take the global state s and the learnable parameters as inputs. In this context, we employ the same mixing network M2 to combine the local values and advantages. However, our framework is flexible enough to allow the use of two different mixing networks for fot and Atot.
In setting, the relationship between the global Q-function, value, and advantage functions is described in Eq. 5 Specifically, we have: A (s. a) = r() a) +ylBs (bajj [ptot (s') - t)tt(s). (s).
To capture this relationship, we train the Q-function by optimizing the following MSE loss:
ming Σ Emsast (s, r(2(s) a) + + yuta((s) not (s))
vot((s))
This is equivalent to: ming, Cg(())) = E (MalaaooMMMlssaaaa - -
(2 a) + My iis' [ Mg
(10)
For the primary loss function used to train the value function, we leverage transitions from the offline dataset to approximate the objective L. resulting in the following loss function for offline training:
(as M Ma a; v w(ss (11)
E(v,, = -m, (so: )1 (ssaa-D
a
As mentioned, after obtaining (V*,07 0*) by solving ming.. L( : 0). we compute the occupancy ratio: woot (s, a) max { f
MMi (9)] MMn- (g(s.a:)..)) }} To train the local policy T=(a:(ss), we
represent it using a policy network n; (s: my)), where 7): are the learnable parameters. The training process involves optimizing the following weighted behavioral cloning (BC) objective:
max L. Eisayo tot (s. a) log( n. (a, n)))
(12)
Our ComaDICE algorithm consists of two primary steps. The first step involves estimating the occupancy ratio fot from the offline dataset. The second step focuses on training the local policy
In our experiments, we single-layer mixing network due to its superior performance compared to two-layer structure, though our approach is general and handle any multi-layer feed-forward mixing network.
7

---

Published as a conference paper at ICLR 2025
by solving the weighted BC problem using w*ate. In the first step, we simultancously update the Q-functions Vg. the mixing network parameters 8, and the value function dh,, aiming to minimize the mean squared error (MSE) in Eq. T0 while optimizing the main loss function in Eq. IT
is important to note that, in practical POMDP scenarios, the global state s is not directly accessible during training and is instead represented by the joint observations from the agents. For notational convenience, we use the global state s in our formulation; however, in practice, it corresponds to the joint observation (s).. Specifically, terms like p (s. a) and pot(ss (s) actually refer to pM a) and y fot (o), where o 2()).
EXPERIMENTS
6.1 ENVIRONMENTS
We utilize three standard MARL environments: SMACv1 (Samvelyan et al. 2019). SMACv2 Ellis al. 2022), and Multi-Agent MuJoCo (MaMujoco) (de Witt et al. 22220. each offering unique challenges and configurations for evaluating cooperative MARL algorithms.
SMACv1. SMACvl based on Blizzard's StarCraft II. It uses the StarCraft API and DeepMind's PySC2 to enable agent interactions with the game. SMACv1 focuses on decentralized micromanage- ment scenarios where each unit is controlled by an RL agent. Tasks like 2c_vs_642g and 5m_ _n_m are labeled hard, while 6h_vs 82 and corridor are super hard. The offline dataset, provided by |Meng] ct al. (203)), was generated using MAPPO-trained agents
Yu et al. 2022).
SMACv2. In comparison to SMACv1, SMACv2 introduces increased randomness and diversity by randomizing start positions, unit types, and modifying sight and attack ranges. This version includes tasks such protoss, terran, and zerg, with instances ranging from 5_vs_5 to 20_rs_23 increasing in difficulty. Our offline dataset for SMACv2 was generated by running MAPPO for 10 million training steps and collecting 1,000 trajectories, ensuring medium quality but comprehensive coverage of the learning process. To the best of our knowledge, we are the first to explore SMACv2 in offline MARL, whereas most prior work has used this environment in online settings.
MaMujoco. MaMujoco serves as a benchmark for continuous cooperative multi-agent robotic control. Derived from the single-agent MuloCo control suite in OpenAl Gym (Brockman al. 2016). it presents scenarios where multiple agents within single robot must collaborate to achieve tasks. The tasks include Hopper-v2, Ant-v2, and HalfCheetah--22 with instances labeled as expert. medium, medium-replay, and medium-expert. The offline dataset was created by (Wang et al. 2022b) using the HAPPO method (Wang et al. 2022a).
6.2 BASELINES
We consider the following baselines, which represent either standard otate-of-the-art (SOTA) methods for offline MARL: (i) BC (Behavioral Cloning); (ii) BCQ Batch-Consttrined Q-learning) (Fujimoto et al. 2019)- an offline RL algorithm that constrains the policy to actions similar to those in the dataset to reduce distributional shift, adapted for offline MARL settings; (iii) CQL (Conservative Q-Learning) (Kumar al., 2020) method that stabilizes offline Q-learning by penalizing out-of-distribution actions, ensuring conservative value estimates; (iv) ICQ (Implicit Constraint Q-learning) (Yang et al. 202T an approach using importance sampling to manage out- of-distribution actions in mulli-agent settings; (v) OMAR (Offline MARL with Actor Rectification) (Pan al., 2022) a method combining CQL with optimization techniques to ensure the global validity of local regularizations, promoting cooperative behavior; (vi) OMIGA (Offline MARI with Implicit Global-to- Local Value Regularization) (Wang et al., 20226) SOTA method that transforms global regularizations into implicit local ones, optimizing local policies with global insights; (vii) OptDICE a naive extension of the OptDICE algorithm| Lee et al. (202I) to multi-agent settings where the global value function directly learned without value factorization; and (viii) AlberDICE Matsunaga et al. (2023) an offline MARL algorithm which also leverages the DICE framework to address the OOD.
We used experimental results contributed by the authors of OMIGA Wang et al. 2022b) as our baselines. They provided both the results and source code for all the baseline methods. This source
8

---

Published as a conference paper at ICLR 2025
Instances
BC
BCQ
CQL
ICQ
OMAR
OMIGA
OPIDIEE
AlberDICE
ComaDICE (ours)
poor
0.0
0.6
2e.ss_iilzg
medicm
1.
2.6
2.5
1.4
1,5
6.25666
1.4
1 1.6
8.8 7.0
good
312299
1..6
444
13.0
28.7
28.7
10.6
37.5
4.22
55.0 1,5
poor
1.5
1.5
0.0:000
0.0
44142
5m.vs.6m
medium
1.
1.5
2.2
1.2
0.
2.1
0.0
0.1
7.5
good
2.3
1.5
191
3.8
3.2112
1.21.
7.3
3,9 1.4
8.1 3.2
poer
1.5
1
6h.vs.8z
medium
1.
1.5
19+ 1.5
2.5122
19
1.2
0.0+0000
2.6
3
good
8.8
8.8 3.6
7.5 1.5
9.4
0.6 0.3
5.6
0.0
0.0
11.2 5.4
poer
corndar
medium
2..
23.1 1.5
144+15
22.3
1.
23.8
19.8
9.4
27.3 3.4
good
3..6
4.4
56
42.5
3.1
4..9
39.6
43.1
48.8 a 2.5
Table I: Comparison of average winrates for ComaDICE and baselines on SMACvI tasks.
OptDICE
AlberDICE
ComaDICR
BCQ
CQL
ICQ
OMIGA
(ours)
5.vs.5
36.9+8.7
16.2+2.3
10.044.1
36.949.1
21.244.1
33.1+5.4
108&11.
12.6=0.9
46.2:6.1
10
36.2-10.6
9.45.66
26.247.6
28.146.6
13.8+7.0
40.++0.77
9.5=0.8
11.8+0.9
50.6
Protoss
10
19.4:4.6
10.044.1
10.645.4
12.544.4
12.543.4
16.246.1
10.040.5
9.840.3
20.0:4.22
20 vs 20
37.524.4
6.242.0
11.924.1
32.548.1
23.842.5
36.245.1
10.022.0
10.120.6
47,547.8
20.v8.23
13.8
1.2:1.5
0.0:0.0
12,545.6
11.247.8
12.548.1
8.121.4
8.8
13,8+5.8
5.vs.5
30.0a4.2
12.5+6.2
9.47799
23.1
14.4a4.7
28. 1+4.4
64s1.1
8.141.4
30.6
10 10
29.445.8
69+6.1
9.445.6
16.9458
15.044.6
29.4+3.2
6.0=166
8.2+1.0
32.5+5.8
Teran
10.vs.11
16.243.6
3.8:4.6
7.5664
5.0:4.2
9
12,545,2
4.8=1.2
6.2+0.9
19,4+5,4
20_vs_20
26.2-10.4
5.0+3.2
10,6+4.2
15.6+3.4
7.5+7.3
21,944.4
6.3+1.8
5.9+1.2
29, 4+.3.
20
7.5+6.1
4.462.5
5.vs.5
26.9=10.0
14.4s4.2
14.4+5.8
18.847.1
13,816.1
21,945.9
8.2s1.8
9.5a008
31,247.77
10 vs: 10
25.0+2.8
5.621.6
5.624.66
15.647.4
19.422.3
23.816.4
7.8=1.0
8.520.3
33.8+11.8
Zorg
10
13 834.7
9.455.2
6.224.4
10.646.7
10.643.8
13,846.7
7.2=0.7
9.1+005
19,4+:366
vs.
10.0+7.8
10.042.3
7.3+0.7
9,4+6.2
20.v8.23
7.5+3.2
0.6
1.251.5
7.5+3.2
3.8=2.3
4.4a4.2
7.1=1.2
8.840.5
11.2a4..
Table 2: Comparison of win rates for ComaDICE and baselines across SMACv2 tasks.
code was also employed to these baselines for the SMACv2 environment. All hyperparameters were kept at their default settings, and each experiment was conducted withfive different random seeds ensure robustness and reproducibility of the results.
6.3 MAIN COMPARISON
We now present comprehensive evaluation of proposed algorithm, ComaDICE, against several baseline methods in offline MARL. The baselines selected for comparison include both standard and SOTA approaches, providing robust benchmark to assess the effectiveness of ComaDICE.
Our evaluation focuses on two primary metrics: returns and winrates. Returns are the average rewards accumulated by the agents across multiple trials, providing a measure of policy effectiveness. Winrates, applicable in competitive environments such as SMACv and SMACv2, indicate the success rate of agents against opponents, reflecting the algorithm's robustness adversarial settings.
The experimental results, summarized in Tables 3 demonstrate that ComaDICE consistently achieves superior performance compared to baseline methods across a range of scenarios. Notably, ComaDICE excels in complex tasks, highlighting its ability to effectively manage distributional shifts challenging environments.
6.4 ABLATION STUDY IMPACT OF THE REGULARIZATION PARAMETER ALPHA
We investigate how varying the regularization parameter alpha (a) affects the performance of our ComaDICE algorithm. The parameter is crucial for balancing the trade-off between maximizing rewards and penalizing deviations from the offline dataset's distribution. We conducted experiments with values ranging from {0.01, 0.1, 10, 100), evaluating performance using average winrates across all the SMACv2 tasks and average returns across all the MaMujoco tasks. These results, illustrated in Figure highlight the sensitivity of ComaDICE to different a values. In particular,

---

Published as a conference paper at ICLR 2025
Instances
BCQ
CQL
ICQ
OMIGA
OptDCE
AlberDiCR
ComaDICC
(ours)
77.9 58.0
139.1 313.8
754.7 1 806.3
K59.6 709.5
655,9 120.1
$44.6 556.5
2827.7 62.9
Hopper
m-replay
medium
26.5 2100
20.6
401.3 199.9
152
19544 103.6
8+ 14.0
1189.3 544.3
774.2 1: 191.3
257.8 4.5.3
204.1 +-419
4 213.5
35.3
906.3 242.1
meexpen
$4.3 23.7
64.8 123.3
355.4 1. 373.9
709.0 595.7
400.9 132.5
303.4 515.1
1362.4 a 522.9
= 2021.6
2050.0 11.9
2055.5 1.6
27.7.2+27..0
1896.8 33.77
2056.9 + 5.9
m-replay
950.8 18.88
231.6 1618.3
1016.7 53.5
1105.1 83.9
889 62.6
1012.8 a 80.8
1122,9 61.0
m-cxpen
1020.9 242.7
800.2 a 1621.5
1590.2 a 85.6
1720,3 a: 110.6
1293.2 1 183.1
1750.0 a 23.6
1813.9 68.4
expert
2992.7 629.7
1189.5 1034.5
2955
459.2
3383.6 552.7
2601 461
3356.4 546.9
4082,9 45.7
1016.9
2549.3
237.4
315.55
m-replay
-333.6 152.1
1998.7 693.9
1922.4 612.9
2504.7 83.5
-912.9 1363,9
440.0 $28.00
2855,0 242.2
m--xper
3543.7 1. 780.9
1194.2 1081.0
2834.0 420.3
29/88.5 5189
-2485.88 2338.4
2288.2 1 759.5
3889.7 81.6
Table 3: Average returns for ComaDICE and baselines on MaMuJoCo benchmarks.
60%-
40%
20%
0.010.1
10 100
0.010.1
10
100
0.010.1
10
100
0.01
0.1
10
100
protoss
terran
zerg
Hopper
Ant
HalfChectah
Figure 1: Impact of regularization parameter a on performance in different environments.
observe that ComaDICE achieves optimal performance when around 10, suggesting that the stationary distribution regularizer plays essential role in the success of our algorithm.
In appendix, provide additional ablation studies to analyze the performance of our algorithm using different forms of f-divergence fuunction, as well as comparisons between l-layer and 2-layer mixing network structures. The appendix also includes proofs of the theoretical claims made in the main paper, details of our experimental settings, and other experimental information.
CONCLUSION, FUTURE WORK AND BROADER IMPACTS
Conclusion. In this paper, we propose ComaDICE, principled framework for offline MARL.. Our algorithm incorporates a stationary distribution shift regularizer into the standard MARL objective to address the conventional distribution shift issue in offline RL. To facilitate training within CTDE framework, we decompose both the global value and advantage functions using a mixing network. We demonstrate that, under mixing architecture, the main objective function is concave in the value function, which is crucial for ensuring stable and efficient training. The results of this training then utilized to derive local policies through weighted BC approach, ensuring consistency between global and local policy optimization. Extensive experiments on SOTA benchmark tasks, including SMACv2, show that ComaDICE outperforms other baseline methods.
Limitations and Future Work: of this paper. For instance, we focus solely on cooperative learning, leaving open the question of how the approach would perform in objective, the DICE term is designed reduce the divergence between the learning policy and the behavior policy. As a result, the performance of the algorithm is heavily dependent on the quality of the behavior policy. Furthermore, our algorithm, like other baselines, still requires large amount of data to achieve desirable learning outcomes. Improving sample efficiency would be another valuable area for future research.
There are some limitations that are not addressed within the scope
settings. Additionally, in our training
Developing offline MARL algorithm with stationary distribution shift reg- ularizer can enhance performance in costly real-time tasks like robotics, autonomous driving, and healthcare. It also enables safer exploration and broader adoption in high-stakes settings. However, reliance on the behavior policy means flawed or biased data could degrade performance, reinforcing biases suboptimal behaviors. Additionally, the algorithm, like any Al systems, risks unintended misuse in surveillance or military applications, where multi-agent systems could manipulate environ- ments without proper oversight.
Broader Impacts:
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is supported by the Lee Kong Chian Fellowship awarded to Tien Mai.
ETHICAL STATEMENT
Our work introduces ComaDICE, framework for offline MARL, aimed at improving training stability and policy optimization in complex multi-agent environments. While this research has significant potential for positive applications, particularly in domains such as autonomous systems, resource management, and multi-agent simulations, it is crucial to address the ethical implications and risks associated with this technology.
The deployment of reinforcement learning systems in real-world, multi-agent settings raises con- cerns about unintended behaviors, especially in safety-critical domains. If the policies learned by ComaDICE are applied without proper testing and validation, they may lead to undesirable harmful outcomes, especially in areas such as autonomous driving, healthcare, or robotics. Additionally, bias in the training data simulation environments could result in suboptimal policies that unfairly impact certain agents or populations, potentially leading to ethical concerns regarding fairness and transparency.
To mitigate these risks, we emphasize the need for extensive testing and validation of policies generated using ComaDICE, particularly in real-world environments where the consequences of errors could be severe. It is also essential to ensure that the datasets and simulations used in training are representative, unbiased, and carefully curated. We encourage practitioners to use human oversight and collaborate with domain experts to ensure that ComaDICE is applied responsibly, particularly in high-stakes settings.
REPRODUCIBILITY STATEMENT
In order facilitate reproducibility, have submitted the source code for ComaDICE, along with the datasets utilized to produce the experimental results presented in this paper (all these will be made publicly available if the paper gets accepted). Additionally, in the appendix, we provide details of our algorithm, including key implementation steps and details needed to replicate the results. The hyper-parameter settings for all experiments are also included ensure that others can reproduce the findings under the same experimental conditions. We invite the research community to explore and apply the ComaDICE framework in various environments further validate and expand upon the results reported in this work.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International conference on machine learning, pp. 104-114. PMLR, 2020.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base based offline reinforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drive offline reinforcement learning. arXiv preprint arXiv:2202. / /566, 2022.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933 49469,34446,,, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
01540
The Viet Bui, Tien Mai, and Thanh Hong Nguyen. Inverse factorized q-learning for cooperative multi-agent imitation learning. Advances in Neural Information Processing Systems, 38, 2024.

---

Published as a conference paper at ICLR 2025
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353-18363, 2020.
Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36, 2024.
Christian Schroeder de Witt, Bei Peng, Vierre-Alxxanre Kamienny, Philipp Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint MxXi::2003066009 19, 2020.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating functional knowledge in neural networks. Journal of Machine Learning Research, 10(6), 2009.
Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Fo- erster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2212.0F1 07.42222...7 2022.
Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, pp. 6863-6877 PMLR, 2022.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Ammar Haydari and Yasin Yılmaz. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(1): 11-32, 2020.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. In ECAI, pp. 1148-1155, 2023.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pp. 651-673.
PMLR, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783. PMLR, 2021.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Jongmin Lee. Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference Machine Learning, pp. 6120-6130. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957/ 2022.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39)11-40, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.06643 2020.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya- Ya-Qin Zhang. Proto: Iterative policy regularized affline-to-onlig reinforcement learning. arXiv preprint arX1v:2305.1566, 2023.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic
offline reinforcement learning. Conference on Robot Learning. pp. 1455-1464. PMLR, 2022.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. arXiv preprint arXiv:2402.003345 2024.
Daiki E Matsunaga, Jongmin Lee, Jaeseok Yoon, Stefanos Leonardos, Pieter Abbeel, and Kee-Eung Kim. Alberdice: addressing stationary distribution correction estimation. Advances Neural Information Processing Systems, 36:72648-72678 2023.
joint actions in offline multi-agent via alternating
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprini 2rXiv:2006.03647, 2020.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang. Yaodong Yang. al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-248, 2023.
Ofir Nachum and Bo Dai. Reinforcement learning via rockhel- duality, arXiv preprint arXiv:2001.018666 0666 2020.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint UrXiv:2006.09359, 2020.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When trust your simulator: Dynamics-awar aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-36612, 2022.
Frans Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-355 2008.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International conference on machine learning, pp. 17221-17237. PMLR, 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weightee regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 11X10:1910.00177,, 2019.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement 2020.
learning. The Journal of Machine Learning Research, 21(1):7234-722
Mikayel Samvelyan. Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint /rXiv: 1902.00044 2019.
13

---

Published as a conference paper at ICLR 2025
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conserva- tive q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Harshit Sikchi, Amy Zhang, and Scott Niekum. Imitation from arbitrary experience: A dual unification of reinforcement and imitation learning methods. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Kyunghwan Son, Daewoo Kim, Wan Kang. David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang. Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226-237, 2022.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex ducling multi-agent q-learning. arXiv preprint arXiv:2008.0100 010622 2020.
Jun Wang, Yaodong Yang, and Zongqing Wang. Trust region policy optimization in multi-agent reinforcement learning. arXiv preprint arXiv:2109. 1/251, 2022a.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems, 36, 2022b.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1361, 2019.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft
behavior regularization. arXiv preprint arXiv:21 10.0035, 2021.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:4085-4098, 2022a.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weigee weighted offline imitation learning from suboptimal demonstrations. In Proceedings of the 39th International Conference on Machine Learning, pp. 24725-24742, 2022b.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022c.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.1588 2023.
Yigin Yang, Xiaoteng Ma. Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 143102 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 15:24611-24673 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 1411124-14442 2020.
14

---

Published as a conference paper at ICLR 2025
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-2888 289677, 2021.
Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 9022-9030, 2022.
Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang. Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offline safe reinforcement learning. arXiv preprint arXiv:2301.1220: 2023.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropp multi-agent reinforcement learning. In International conference on machine learning, pp. 12491-12500. PMLR, 2021.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li. Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guie guided diffusion model. arXiv preprint 20X102240..17000, 2024.
15


Reviews:
[{"review_id": "oGBt47piUj", "reviewer": "Reviewer_RYfn", "summary": "This paper proposes an algorithm introducing stationary distribution correction to address the distributional shift problem in offline cooperative multi-agent reinforcement learning (MARL). In multi-agent environments, this issue is intensified due to the large joint state-action space and the interdependencies among agents. To tackle this, ComaDICE minimizes the f-divergence between the stationary distributions of the learning and behavior policies. Additionally, by leveraging the Centralized Training with Decentralized Execution (CTDE) framework, it decomposes the global value functions into local values for each agent, ensuring that the optimization of each agent’s local policy is consistent with the global learning objective.", "questions": "The ablation study shows that performance decreases when using a more complex network. Why is that the case? If more data were available or the model was improved, could the results potentially differ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Effective Distribution Correction in Multi-Agent Settings: ComaDICE improves upon traditional DICE by extending stationary distribution correction to multi-agent environments. Through f-divergence-based alignment between behavior and target policies, it effectively handles the complex distributional shifts unique to multi-agent interactions, enhancing policy reliability and performance.\n- Theoretical Foundation: A thorough mathematical analysis provides convergence and stability proofs through f-divergence correction, reinforcing the algorithm’s reliability in multi-agent scenarios.\n- Stable Value Decomposition: ComaDICE decomposes global values into convex local objectives, enhancing training stability and aligning local agent optimization with the global objective to address coordination and stability challenges specific to multi-agent environments.", "weaknesses": "- ComaDICE’s theoretical analysis relies on non-negative weights and convex activations in the mixing network for stability. While this aids convergence, it may limit the model’s ability to capture complex inter-agent dynamics. In the practical algorithm, this limitation is intensified by the use of a single-layer linear mixing network, which further restricts representational capacity in highly interactive environments.\n- High Computational Cost: ComaDICE incurs a significant computational cost due to the precise f-divergence-based adjustments needed for each agent’s local policy in stationary distribution correction. This can lead to reduced efficiency, especially in environments with a large number of agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UdNt5ojj0y", "reviewer": "Reviewer_kymz", "summary": "The paper proposes ComaDICE, a stationary distribution correction estimation approach to addressing OOD states and actions in Offline MARL. The derivation starts with the LP formulation of RL in the joint state-action space, which results in a concave objective for learning state-value functions similar to OptiDICE [2]. The objective also uses value decomposition with a monotonic mixing network for the advantage function similar to OMIGA [3]. ComaDICE is evaluated on SMACv2 and MaMuJoCo and shows comparable performance with baselines such as BC and OMIGA.", "questions": "1. What is the main purpose for the value decomposition and learning individual Q-functions? \n\n1. Related to the first question, what is the purpose of Theorem 4.2, if the original loss is also concave? If we use the mixing functions, does $ \\mathcal{\\tilde L}$ still find the global optimum?\n\n1. Can ComaDICE solve the XOR game and Bridge mentioned in AlberDICE? \n\n1. Related to “Strengths”, can solving ComaDICE learn the global optimum in the underlying MDP even with the mixing network?\n\n\n1. Is the Individual Global Max (IGM) assumption required in order to introduce the mixing network? In other words, does ComaDICE assume that the underlying optimal Q function assume IGM?\n\n1. What is the main difference in the final algorithm with OMIGA [3] ?\n\n1. Why are the notations mixed between using joint observations $\\mathbf{o}$ and $s$? Is the setting a Dec-POMDP or a specialized setting where the joint observations constitute the state?\n\n1. Please write in a different color during the rebuttal which part is the novel part. In particular, which part of the proofs are a novel contribution and which part is coming from OptiDICE [2]\n\n1. Please address both my comments in the Strengths and Weaknesses section in detail.\n\n\n### References\n[1] AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation (Matsunaga et, al. NeurIPS 2023)\n\n[2] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation (Lee et, al. ICML 2021)\n\n[3] Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization (Wang et, al. NeurIPS 2023)\n\n----------------------------------------------------------------------------------------------------------------------\nPost-Rebuttal\n----------------------------------------------------------------------------------------------------------------------\n\nDuring the rebuttal, the authors worked hard to address my concerns, namely (1) additional baselines (AlberDICE and OptiDICE) on all 3 environments, (2) additional toy example (XOR Game), (3) a re-interpretation of the method as learning the optimal policy implicitly in factorized policy space and (4) detailed analysis on how ComaDICE works based on the XOR Game which is a toy domain. \n\nThe reason why my initial score was low was because a lot of these points were not explicit in the original draft, and it was unclear what the contribution was. After the rebuttal and some discussion, these points were made clear and the authors demonstrated that (1) using a factorized approach to decomposing stationary distributions is implicitly searching for optimal factorized policies (2) ComaDICE is able to approximate a global optimum as opposed to Nash policies (AlberDICE), (3) ComaDICE outperforms all baselines in all environments including AlberDICE and OptiDICE, which are most closely related, (4) the algorithm is more scalable and simple compared to AlberDICE.\n\nAs it currently stands, these contributions of the paper are scattered and it would take some effort by the reader to appreciate the paper, without some background in both DICE-based approaches in MARL. I would recommend a re-write of the storyline of the paper, highlighting the fact that ComaDICE pushes the cooperative Offline MARL field forward beyond IGM as well as AlberDICE.  \n\nAs a result, I've increased my score as follows:\n\nSoundness: 2-->3\n\nPresentation: 1-->2\n\nContribution: 1--> 3\n\nOverall Score: 3: reject, not good enough --> 6: marginally above the acceptance threshold\n\nI can't quite give an 8 (Strong Accept) due to the writing, but I would have given 7 if that option was available.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "It is known that every MDP has a deterministic optimal policy, which, if extended to the multi-agent setting, can be factorized into independent policies. A very optimistic reading of the paper would interpret that the decomposition procedure is actually learning optimal value functions over decentralized policies. If this interpretation is correct, the ComaDICE presents a very scalable and principled offline MARL algorithm for learning decentralized value functions and policies without any restrictive IGM assumption as in previous work such as ICQ. However, it is worth noting that this is more of a statement of the potential of the paper rather than a strength of its current version, as these points are not specifically addressed in the current draft.\n\nFurthermore, ComaDICE can be more scalable in comparison to AlberDICE [1] which requires alternating optimization.", "weaknesses": "### Problem Statement\nIt is not clear what the main problem is that ComaDICE is solving. It is briefly mentioned in the introduction that OOD states and actions are a problem in offline MARL. However, this was addressed in detail by other work such as CFCQL/OMIGA/AlberDICE mentioned in the Related Work (especially AlberDICE [1] which is the most similar). For instance, AlberDICE considers some coordination problems (XOR/Bridge/etc.) where OOD joint actions may be common, these is no consideration of these settings as well as comparison to AlberDICE both algorithmically and empirically. Thus, it is not entirely clear from the current draft of the paper what the main problem is that ComaDICE is solving, and if it is indeed OOD actions, which part of the algorithm in particular is alleviating this.\n\n### Novelty\nThe derivation is based on OptiDICE [2] but this is not explicitly mentioned, which is misleading. \nFor instance, Proposition 4.1 seems equivalent to Proposition 1 of OptiDICE. Furthermore, an extension of OptiDICE to solve Offline MARL was considered in AlberDICE [1] so it is not clear why ComaDICE should be preferred over AlberDICE. Furthermore, the final algorithm closely resembles OMIGA [3].\n\n### Algorithm\nIt is unclear what the purpose of value decomposition in learning the Q functions is. It seems the advantage can be computed by the learned state-value function $\\nu$ and run WBC (as mentioned in Appendix C of AlberDICE[1]). Also, Line 346 defines $A_\\nu^{tot}$ as the sum of reward and state-value functions.\n\n### Lack of Relevant Baselines\nAlberDICE and OptiDICE are missing as the main baselines, as well as CFCQL which addresses OOD actions but in a different manner. These are all mentioned in the Related Work section but not compared.\n\n### Writing\nThe paper is generally not well-written. All of the aforementioned weaknesses of the paper should be addressed in detail and the writing should not raise any of these concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLsj7E5Y7h", "reviewer": "Reviewer_3fo7", "summary": "In this paper, ComaDICE algorithm is proposed for offline multi-agent reinforcement learning, extending the DICE method to multi-agent scenarios and using value function decomposition and policy extraction methods, the idea is novel and the experimental results show the potential of the method.", "questions": "* The discussion of the DICE method and its goals in the Related Work and Preliminary Knowledge section is not sufficiently in-depth to clearly locate the innovations of this paper with respect to existing work.\n* The proof section of DICE skips some key steps,  has a confusingly organized derivation and notation that does not clearly state the purpose and rationale for each step of mathematical transformation. Interpretation between Lagrange functions and offline reinforcement learning problem formulation is in lack.\n* The paper fails to explore the common sparse reward scenario in offline reinforcement learning, and the analysis of the quality requirements of behavioral strategies is insufficient, limiting the feasibility of the method in practical applications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The introduction of the DICE algorithm within the domain of multi-agent reinforcement learning represents a significant contribution characterized by its novelty and innovative approach.\n* The experimental settings are comprehensive, demonstrating the potential of the proposed algorithm effectively.", "weaknesses": "* The contribution emphasizes the value decomposition and claims to prove the equivalence of the local policy product to the globally optimal policy, but the paper is too repetitive and superficial in its exposition of the decomposition method, neither explicitly defining the local subtasks and their corresponding policies, nor the relationship between them, nor providing a complete proof of this equivalence.\n* The experimental results for hybrid networks contradict mainstream research findings (single-layer outperforms double-layers), a phenomenon that hints at a possible fundamental flaw in the application of the DICE methodology to MARL, but the paper lacks an in-depth discussion of this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dkESPrdGU7", "reviewer": "Reviewer_XXjk", "summary": "This work introduces ComaDICE (offline Cooperative MARL with DICE), an approach for offline cooperative multi-agent reinforcement learning (RL) that leverages the DICE method. ComaDICE formulates the offline cooperative multi-agent RL problem as a constrained optimization and employs a DICE-based method to compute a global Lagrangian multiplier, $\\nu^{tot}$. Given the large state and action spaces typical in multi-agent RL, practical optimization of $\\nu^{tot}$ may not be feasible. To address this challenge, ComaDICE employs value function decomposition to decompose the global Lagrangian multiplier into individual Lagrangian multipliers for each agent.", "questions": "Q1. Do the authors assume that $s$ can be sufficiently represented from $o$?\nNotations for states and observation are confusingly used. For example, in line 234 and 237, $\\nu(s)$ and $q(s,a)$ are defined as a collection of functions that requires individual observations,$\\nu_i(o_i)$ and $q(o_i,a_i)$ respectively If so, the assumption should be explicitly clarified in the POMDP description.\n\nQ2. Figure 2 lacks information on which task was selected for each benchmark. (e.g. 5_vs_5 or 10_vs_10 in protoss, or expert or medium data quality in Hopper) Could you clarify?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work provides a comprehensive performance evaluation through experiments conducted on an extensive set of benchmarks, including the challenging multi-agent RL benchmark SMAC-v2. The experimental results indicate that the proposed approach shows superior performance relative to the baselines considered in the manuscript.", "weaknesses": "The primary concern regarding this work is its novelty compared to previous work.\n\nSpecifically, AlberDICE by Matsunaga et al. (2023) [A] conveys a similar idea, i.e. adoption of DICE for offline cooperative multi-agent RL. The key difference of ComaDICE from AlberDICE appears to be employing individual Langrangian multipliers $\\nu_i$ and the mixing network for value function decomposition: while AlberDICE utilizes a simple yet principled resampling method for obtaining $\\nu_i$, ComaDICE employs the value factorization as discussed in Section 4.2, which requires the reliance on the additional mixing network $\\mathcal{M}_\\theta$ further necessitates additional training.\n\nAlthough AlberDICE is briefly mentioned in Section 2, the paper does not adequately discuss the theoretical or empirical advantages of adopting the value decomposition instead of the resampling approach. In addition, the AlberDICE paper presents a simple multi-agent task where value function decomposition leads to a substantial performance loss.\n \nThe absence of such a comparison significantly weakens the perceived contribution of this study, as it fails to establish a clear improvement of differentiation from previous work. In this sense,\n(1) A detailed discussion comparing ComaDICE and AlberDICE should be added,\n(2) AlberDICE should be included as a baseline in all experiments,\n(3) ComaDICE should be tested on the XOR game in the AlberDICE paper  \n\n\n[A] Matsunaga et al., “AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation.”, NeurIPS 2023.\n\n[Minor comments]\nIn Sections 6.1 and 6.3, SMACv1 environment is discussed; however, corresponding experimental results for SMACv1 are not included in the main manuscript. It would be more appropriate to relocate these discussions to the appendix and direct readers there for further details, or incorporate the result on SMACv1 to the main manuscript, potentially replacing some of the redundant results on SMACv2 in Table 1 or Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)