Who Matters Matters: Agent-Specific Conservative Offline MARL
Authors: Haosheng Chen
Yun Hua
Wenhao Li
Shiqin Wang
Xiangfeng Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Offline reinforcement learning
Reinforcement learning
Primary area: reinforcement learning

Abstract: Offline Multi-Agent Reinforcement Learning (MARL) enables policy learning from static datasets in multi-agent systems, eliminating the need for risky or costly environment interactions during training. A central challenge in offline MARL lies in achieving effective collaboration among heterogeneous agents under the constraints of fixed datasets, where \textbf{conservatism} is introduced to restrict behaviors to data-supported distributions. Agents with distinct roles and capabilities require individualized conservatism - yet must maintain cohesive team performance. However, existing approaches often apply uniform conservatism across all agents, leading to over-constraining critical agents and under-constraining others, which hampers effective collaboration.
To address this issue, a novel framework, \textbf{OMCDA}, is proposed, where the degree of conservatism is dynamically adjusted for individual agents based on their impact on overall system performance. The framework is characterized by two key innovations: (1) A decomposed Q-function architecture is introduced to disentangle return computation from policy deviation assessment, allowing precise evaluations of each agent's contribution; and (2) An adaptive conservatism mechanism is developed to scale constraint strength according to both behavior policy divergence and the estimated importance of agents to the system.
Experiments on MuJoCo and SMAC show OMCDA outperforms existing offline MARL methods, effectively balancing the flexibility and conservatism across agents while ensuring fair credit assignment and better collaboration.

Published as a conference paper at ICLR 2026
WHO MATTERS MATTERS: AGENT-SPECIFIC CONSER- VATIVE OFFLINE MARL
Haosheng Chen', Yun Hua?. Wenhao Li Shiqin Wang', Xiangfeng Wang4.5.5
School of Computer Science and Technology. East China Normal University
2. Antai College of Economics and Management, Shanghai Jíao Tong University
3. School of Computer Science and Technology, Tongji University
4. Key Laboratory of Mathematics and Engineering Applications (MoE), East China Normal University
5. Shenzhen Loop Area Institute (SLAI)
hs machensstu.c ser.uesuuuuuuu...... ecnu. edu. en, hyyh28esjtu. edu. cn, whli@tongji edu. en, 512759011378st ecnu.
ecnu. edu. en, xfwangecs. ecnu.edu.
ABSTRACT
Offline Multi-Agent Reinforcement Learning (MARL) enables policy learning from static datasets multi-agent systems, eliminating the need for risky or costly environment interactions during training. A central challenge in offline MARL lies in achieving effective collaboration among heterogeneous agents under the con- straints of fixed datasets, where conservatism is introduced restrict behaviors data-supported distributions. Agents with distinct roles and capabilities require indi- vidualized conservatism yet must maintain cohesive team performance. However, existing approaches often apply uniform conservatism across all agents, leading to over-constraining critical agents and under-constraining others, which hampers effective collaboration. To address this issue, novel framework, OMCDA, is proposed, where the degree of conservatism is dynamically adjusted for individual agents based their impact overall system performance. The framework is characterized by two key innovations: (1) A decomposed Q-function architecture is introduced to disentangle return computation from policy deviation assessment, allowing precise evaluations of each agent's contribution; and (2) An adaptive conservatism mechanism is developed to scale constraint strength according both behavior policy divergence and the estimated importance of agents to the system. Experiments on MuJoCo and SMAC show OMCDA outperforms exist- ing offline MARL methods, effectively balancing the flexibility and conservatism across agents while ensuring fair credit assignment and better collaboration.
INTRODUCTION
Multi-agent reinforcement learning (MARL) has gained significant traction in domains such as autonomous driving (Cao et al. 2012), collaborative robotics (Orr & Duttal 2023). and multi-player games (Berner et al. 2019 et al. 2025), where agents must learn to coordinate or compete to accomplish complex objectives. Despite its successes, most MARL approaches assume agents can interact freely with the environment during training. In practice, however, this assumption often breaks down due high interaction costs, safety concerns, operational constraints (Wang et al.) 2024). To address these limitations, Offline Reinforcement Learning (Offline RL) has emerged as a compelling alternative (Fujimoto & Gui [021]] Kostrikov al., 2021b]] Kumar al. 2020 Levine et al. 2020; Wu et al., 2019). Instead of relying on real-time interactions, Offline RL learns from pre-collected datasets, making it more practical for safety-critical or data-scarce environments. In the single-agent setting, Offline RI. has achieved notable progress by addressing challenges such as Q-value overestimation for out-of-distributiin (OOD) actions, which often lead to poor generalization. A key development in this direction is the use of conservative methods (Wu et al., 2019), which penalize unlikely or unsupported actions to ensure that learned policies remain close to the behavior policy. This form of conservatism is defined as the tendency favor actions supported by the training data while avoiding uncertain or OOD behaviors which improves stability and robustness during offline learning (Kumar et al. 2020

---

Published as a conference paper at ICLR 2026
When Offline RL is extended to multi-agent settings (Offline MARL), the situation becomes more complex. The interplay among agents introduces increased non- non-stationarity. and the offline dataset can exhibit more severe distributional shifts. Moreover, credit assigmeent-how each agent's actions contribute to overall joint performance-presee
a substantial challenge (Wang & Zhan 2023 Yang et al. 202T). Recent efforts has explored Offline MARL under the "Centralized Training and Decentralized Execution" " (CTDE) framework (Lowe et al. 2017). leveraging multi-agent value decomposition combined with offline conservatism stabilize learning.
Nevertheless, existing studies seldom consider the heterogeneity of agents in real world applications. Due to their distinct roles and interaction patterns, different agents can wield unequal influence on overall system performance (Wang et al. 20206] Foerster et al. 2018). For instance, in a football team, strikers are often encouraged to take creative, high-risk actions maximize scoring opportunities, while defenders must adhere to disciplined, risk-averse strategies to ensure team stability. Imposing equal conservatism on both roles would limit the striker's effectiveness and increase the defender's exposure to costly errors. This illustrates that the appropriate level of conservatism should depend on the agent's 's role, uncertainty, and potential impact. Consequently, central challenge in heterogeneous Offline MARL is how to adaptively assign conservatism across agents based their individual characteristics -ttiiiiiiiissiiiiiinggg striking a balance between safety and exploration that enhances both performance and reliability as promote their collaboration.
(U20stFoerrser
In this study, we introduce a novel offline MARL approach, Offline MARL with Conservative Degree Allocation (OMCDA), which addresses the challenge of distributing conservatism among agents based on their deviations from behavior policies and their impact on system performance for heterogeneous agents in offline MARL. OMCDA decomposes the Q-function in offline MARL with regularization into two components: one for computing the return and the other for capturing policy deviations. This decomposition isolates the impact of deviations, enabling clearer and more accurate learning process. The conservative degree of each agent is dynamically adjusted based on the effect of their deviations on the overall return, promoting a balanced influence on system performance. This dynamic allocation is integrated into the OMCDA framework, ensuring a balance between conservatism and flcxibility, and consistent credit assignment to enhance teamwork.
The key contributions of this paper are as follows: (1) A comprehensive analysis of conservative degree allocation in heterogeneous offline MARL, exploring how varying conservative degrees affect individual agent returns and overall system performance. offline MARL algorithm that dynamically adjusts each agent's 's conservative degree based on its impact on system performance, balancing conservatism and flexibility while ensuring consistent credit assignment and promoting collaboration. (3) Extensive experiments on diverse datasets,
(2) The introduction of OMCDA, novel
including multi-agent MuJoCo (de Witt al., 2020) and the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019). showing that OMCDA consistently outperforms existing methods across different environments and datasets.
PRELIMINARIES
We consider a MARL problem following (Wang et al. 2024) where the environment is modeled as a multi-agent Partially Observable Markov Decision Process (Boutilier) 1996), defined by the tuple: G = AS, r, Z, O, n, 7). E S is the true state of the environment. A denotes the action set for each of the n agents. At every time step, each agent i 2........) ,.) chooses an action a, A, forming joint action a = (01, Si.,a..... an) E A". It causes a transition to the next state s' in the environment according to the transition dynamics P(s'|s, a) : S x A" S [0, 1]. All agents share the same global reward function r(s, a) S x A + IR. E [0, 1) is discount factor. In the partially observable environment, each agent draws an observation o, E at each step from the observation function is S N + O. The objective of the team is to learn set of policies π = (T), .2, ) that collectively maximize the expected discounted cumulative reward of the entire system. In the offline setting, agents do not interact with the environment directly but instead learn policies from a static dataset D containing state-action-re reward tuples. The challenge lies in learning effective policies without additional environment interactions.
The Centralized Training with Decentralized Execution (CTDE) framework is widely used in cooperative multi-agent reinforcement learning (MARL) (Oliehoek al. 2008). In
CTDE Framework
2

---

Published as a conference paper at ICLR 2026
CTDE, agents are trained centrally with global information, enabling coordinated policy learning
(Lwe et al.) 2017). During execution, they act based on decentralized local observations, ensur-
ing scalability
real-world settings. A key approach in this framework is value decomposition (Rashid et al. 2020; Sunehag et al. 2017 Wang et al. 2020a), where the global value function is
factorized into local components for each agent. Algorithms such QMIX (Rashid et al. 2020
and VDN (Sunehag et al., 2017) employ monotonic value decomposition for scalable multi-agent
learning. This framework has been adopted in offline MARL (Pan al. 2022
Yang al. 2021
with the global-local Q-value relationship:
Qtor(o, a)
Σ (,(o))i(oraaa)) + b(0), wi Vi 1...n.
where wi; (o), b(o) are local function weights/bias and 0z 0g agent actions/observations.
Offline MARL with Policy regularization Policy regularization constrains policy learning to remain close the behavior policy (Xu et al. 2023), preventing -ittr--diitribution actions that could degrade performance. Several offliee RL algorithms (Wu et al.] 2019) Xu et al. 2023) use this approach to mitigate distributional shift. For example, BRAC (Wu et al. 2019) regularizes the actor's policy to stay close to the behavior policy while optimizing the critic with the standard value function update. In offline MARL settings, this regularized method can be extended, with the objective written as:
max, E
Ea-x [f (I (ax or), mbb (az o+))] S e.
Here π = (TI,.... TN), f is a divergence term that quantifies how far the policy deviates from the behavior policy The a and o, are the action and state timestep while is the constraint of f. We then convert the constrained optimization problem above into an unconstrained one using Lagrangian relaxation by introducing a penalty hyperparameter a::
max E
Σ
(r: (r+
( a (at o.). To (ot o.)))
In this paper, we use Kallback-Leiblee (KL) (eerggence(Pére-Ceee 2008) expressed as DKL to constrain the learning policy and behaviour policy, while the Q-function can be formulated as:
(o.
E
r (re a (xL ll mb))
where n, and r, are the policy and reward at timestep t. To address the issue of conservative degree allocation, provide different levels of conservatism to agents in offline MARL by assigning each agent an individual conservative degree d, which defines the permissible range deviation from its behavior policy. The problem is then reformulated as the following:
max; E DI 7're). s.t. Easx log 1oa (a(o)o))
< di, =,didrot Vi 1 ...n.
(5)
Where d, is the local conservative degree, and de is the global conservative degree which is fixed
value. A deeper analysis of Eq. (5). which reveals the origin of the deviation term in Eq. 12). (2).1. is
provided in Appendix E.5 Then similar to the process of transitioning from Eq. (2) to Eq. (3). we can convert Eq. (5) and assign conservatism level each agent i, denoted as as, while the current Q-function can be formulated as:
r2 1 - a: ( |l ma)
(0, =
In the next section, OMCDA is introduced, built upon the decomposition of the Q-function and dynamic conservative degree allocation. We will demonstrate how this framework addresses the challenges of conservative degree allocation and emphasize it's advantages in offline MARL systems.
3

---

Published as a conference paper at ICLR 2026
Influence
Conservative
Agent
m
conservative degres
Total
d1 mh des
d2 m2 dest
Agent
12
IntToence Calculator
deps
Agent
mhn
dy my des
V
log sila, syla, (m))
Bove
Update
Damnee
Constraint
Figure 1: Overview of dynamic conservative degree allocation framework in OMCDA. 1) The influence calculator takes the policy from each agent, along with the returnbasee state-value function V" and the behavior policy иi derived from the data in dataset, input generate the influence for each agent on the system. 2) Each agent's 's conservative degree is then allocated from the total conservative degree based on the influence term. 3) Finally, the conservative degree is integrated as network update constraint, enabling dynamic allocation while ensuring consistent credit assignment.
3 OMCDA
In this section, we present OMCDA for dynamic conservative degree allocation in offline MARL. First, we motivate the problem through simple example, then decompose the Q-function to quantify individual agent contributions. We develop an adaptive mechanism that adjusts each agent's 's conser- vatism according its impact on system rewards. OMCDA ensures conservatism levels align with agents' influence returns, enhancing performance while preserving consistent credit assignment.
Conservative Degree Allocation in Offline MARL In offline MARL, agents' influences on the system are uniform. To fully leverage these influences and improve system performance, dynamic conservative degree allocation is necessary. This approach allows high-impact agents to make larger deviations, enhancing their contribution to the overall performance. To better illustrate this issue, present a 2-player toy example in Table
In this game, we employ mixed strategy, As a cooperative team, individual players cannot access their personal rewards directly; instead, they observe the team's expected reward Tream which is:
Σ Σ
(1 ((a)))a(())((1))))
61 e)E(A. a)E(A.B)
The game features two possible actions (A and B) for each player. Consider both play- ers following uniform behavior policy To
(0.5, 0.5). The offline dataset D is col- lected under Th. containing policy pairs with their corresponding team reward Tieam: D riemm) }} Clearly, Player achieves higher rewards and greater influence team performance than Player 2, justifying greater allowance for policy deviation. We quantify deviation of agent i's policy from behavior policy through Manhattan distance
{(T) (0.5,0.5), T2 = 0.5),, Fieam]]
(Chuu et al. 2016) A; EeELA,B (T((a))
Table 1: Payoff matrix for the 2-player example.
Thehavior (a)
Player 1/2
A
To align with traditional offline methods, the total conservative degree Atsual for the entire system is set to 0.4. Under uniform conservative allocation, both players share the total deviation equally: A = A₂ = 0.2. After training, the optimal strategies for both players are (0.6, 0.4), increasing team's expected reward by 0.1. Under dynamic allocation, Player receives a larger deviation: A = 0.3, reflecting its higher impact, while
A
3,1
3, 1
B
2,1
2,
A2 0.1. Consequently, Player 1 learns aggressive strategy (0.65, 0.35), while Player
4

---

Published as a conference paper at ICLR 2026
remains near the behavior policy (0.55, 0.45). The dynamic allocation improves team's expected reward more by 0.15, demonstrating its effectiveness coordinating heterogeneous agents.
Decomposition Framework We present the decomposition framework for value functions, aiming to assign different conservative degrees to each agent, as described in Eq. (5). To achieve this, it's crucial to understand how agent's 's deviation from the behavior policy affects the overall return. In offline RL with regularization, both the Q-function and value function contain entangled return and constraint components (Eq. 44, complicating the measurement of agent's is influence on the return. Inspired by BOPAH (Lee et al. 2020), our framework disentangles these components by decomposing the Q-function and value function into two parts: one that computes the return and another that accounts for the deviation constraint. In our framework, the original Q-function in offline RL with regularization in Eq. can be written as:
(o.a) Q" (o.0)))) Q" == E
Σ Yr
Q =
Σ En' DKL [b)] (8)
In this definition, Q" (o, a) calculates the return, and Qa (o, a) captures the deviation from the behavior policy. Similar to these two Q-functions, the decomposition of V-function can also be obtained as:
V (o, = V (o,a) t (o.aa)
(9)
Then with the current Q-function and V-function, the corresponding Bellman backup operators is formulated as:
(TF) (o. == r(o, a) + yBoooo (V" (o'), (TF) Q a) == Booa (o'),
(10)
where V-functions is written similar to SAC (Haarnoja al. 2018 as:
(O Ea-r (Q' (* Eavs (o, log T(a 1 o) To(a
(1I)
By decoupling the Q-function into separate return and deviation components, we isolate return calculation from conservatism enforcement. This enables precise assessment of each agent's influ- ence on cumulative returns, free from conservatism constraint interference. This approach proves particularly crucial in offline MARL, where individual actions affect the joint return. When extending to multi-agent case, according to Eq. (6) and the definitions in Eq. (8). the global Q-function (a
detailed analysis of the relationship between Eq. (5) and the deviation term in Eq. (12) is provided in
Appendix E.5 can be derived within the QMIX framework (Rashid et al., 2020 as follows:
N
Qtor (o, = Vior (0,
+
aN Q (o,a),
(12)
where
Pio a) = Σ w[(()) (((())
(13)
wuf (o) = (z),
(0, =)
(14)
In Eq. 13). Vist represents the global return information, which is distributed to individual agents through value decomposition, with w" and b" as the weight and bias parameters for each agent's local return function Q Eq. (14p defines agent i's conservatism value function (= a), computed as weighted
all agents' conservatism values Q5(0j, az). Here, wi²((o) (o) denotes the observation- dependent weight for agent j's contribution to agent i's conservatism, while (o)(() serves as an adaptive bias term. The decomposition of the V-function is derived in the same manner as the Q-function. The decomposed forms of Vra and Voi are expressed as follows:
V (o) = Σ W (o) ('())
(15)
(o) Σ (0
(o)
(16)
With the decomposition framework, each agent can balance both individual and global constraints effectively, while also more accurately assessing both its own and the overall system's return.
5

---

Published as a conference paper at ICLR 2026
Dynamic Conservative Degree Allocation for Agents in Offline MARL Setting After we get decomposition framework, since the goal to maximum the return is equal to maximum global return- based state-value function Viot- The maximum term in Eq.(5) can be changed into max, E (o
Next, we illustrate the approach to develop a dynamic adaptation method (shown in Figure. D that adjusts the conservative degrees for agents dynamically. Let take another look the constraint term in Eq.(5) where we want to adaptively assign a conservative degree d, to each agent. Given the total degree dior, an efficient strategy is to allocate it based on the influence of the agents on the system. Hence, we propose an influence term m, for each agent i, and de can be obtained as:
d m, drot-
(17)
As shown in Table 1 in offline MARL settings, agent's influence on the system determines the sensitivity of the system to its behavioral policy deviations. Thus, we quantify each agent's 's influence the impact of its policy deviation on the collective return Vi..- Taking the expression of Vor in (T. into account, the influence can be derived by computing the partial derivative of the return-based value function V with respect the KL divergence between the agent's current policy T, and its own behavior policy T6 following:
OV (0) m. kD (TT; mi)
(18)
In practice, to facilitate computation, the chain rule is applied to break down complex dependencies between (o) and the KL divergence DKL( mf). enabling efficient influence computation (further details are in Appendix E.2
m, VVia(o) (DDL((T; TTp))
(19)
Om:
OT,
The first term in (9 captures the strategy change's system impact, while the second term acts as a constraint, measuring the agent's deviation from its behavior policy. Since V% isolates the conservatism term from Viot. it directly quantifies how policy deviations affect system returns. This reveals the relationship between an agent's policy deviation and its return impact. (q.(8)) dynamically determines each agent's conservatism constraint, measuring system return sensitivity to policy deviations. A larger derivative indicates greater positive return impact, permitting more flexible d; smaller derivatives warrant stricter constraints to mitigate risk. Due E, d; diot in (5. to determine the appropriate conservative degree d, for each agent, we adopt a softmax function normalize the weights across all agents:
E VV (0)
VV (0) @D (N || TTM
]
m = |m.... m = Softmax
(20)
@D (T)
TT)
After obtaining m. each d, can be derived using Eq. 17). The conservatism level introduced in T2) can be updated according to following objective:
min (a: d, a4 DKL (Te m!))
(21)
With the conservatism levels on obtained for each agent, we apply these dynamic adjustments the offline MARL environment. We begin by deriving the optimal global policy in the offline MARL setting in Proposition 3.1
Proposition 3.1. an offline MARL setting, the optimal global policy mi (a o) is given by Eq. (4) and is formally expressed follows:
Tio (a o) To(a o) exp ( ( (((00)))
(22)
where Q" (o, a) is optimal action-value function, (o) is optimal value function, and a global conservatism level a is assumed that controls the overall deviation from the behavior policy.

---

Published as a conference paper at ICLR 2026
The proof is based the principles of soft Q-learning Haarnjja Halarning(Haarojja al. 2018 and extend it to offline MARL context. Then, we aim to derive the local optimal policy from the global optimal policy in Eq. (22) by applying the individual conservatism level a, for each agent in Proposition 3.2 and demonstrate its validity Theorem 3.3
Proposition 3.2. Joint policy nus decomposed into product of individual agent policies n. as:
N Ttot 0) T.((; oi).
Based on decomposition in 12 n. (a; o.) =mb(a oz) exp
16
the optimal policy (a= os) for each agent is given by:
w (o) (or (on. -V (o.)) + (Q (o. voi (0)
(23)
where a; controls the conservatism level of the agent's policy, and प; denotes the optimal policy that satisfies Eq. 5y.
Theorem 3.3. Given Eq. 23,. the optimal policy for each agent is derived, and consistency between the local optimal policies oh and the global optimal policy Viot is guaranteed. This consistency holds for individual 0x assignments across agents.
The optimal q. is then used to update each agent's ' conservatism- value function V It should be noted that each local policy needs satisfy Ea,wi, ~x, ", (a, 0,) = Therefore, according to Eq. 23). the following formula can be obtained:
m (o) (Ø: (oi, Vi¹ (T:(o))) + (qr (o,a) Vos" (o) (o)) =1 (24)
E Bagasa
exp
Or
Proposition 3.4. From Eq. 24 each agent's conservatisn-basse value function V, updated through the following optimization:
w (0) (Q) (o,, az) VT' (o:))
min E (o
exp
(25)
(Q (0, yoi vo. / (o)V?" (0:) + a, Vo.
The proofs of Proposition 3.1 Proposition 3.4 and Theorem 3.3 are provided in Appendix C
Dynamic conservatism has now been incorporated into MARL frameworks, enabling agents to optimize their behavior in offline settings through adaptive balancing between conservatism and policy deviation- with each agent's contribution weighted by its measured impact on collective system performance. The algorithm and additional explanation of OMCDA is in Appendix E.
Comparison with prior works Prior works including FOP (Zhang et al., 202T), ADER (Kim & Sung 2023), and CFCQL (Shao al. (024) have investigated adaptive approaches in MARL. While FOP and ADER are online methods that employ dynamic entropy regularization similar to OMCDA, their adaptive mechanisms confined to policy updates, applying either global uniform constraints or no constraints to Q/V function updates-an approach that fails to address per-agent constraint allocation for heterogeneous agents, which crucial for mitigating OOD issues in offline MARL. In contrast, OMCDA uniquely enables dynamic conservatism allocation for both policies and value functions, ensuring optimal updates in oflline settings. Offline method CFCQL determines conservatism by behavior policy deviation, while OMCDA considers each agent's impact on system performance. This allows OMCDA to balance conservatism and flexibility, optimizing performance.
EXPERIMENT
In this section, conduct experiments to: (1) evaluate OMCDA's performance, (2) demonstrate its effectiveness in dynamic conservative degree allocation, and (3) analyze key components and choices of total conservative degrees. Further ablation details in Appendix
7

---

Published as a conference paper at ICLR 2026
Mulli-Agent MuJoCoo
SMAC
Figure 2: The average returns for the offline Multi-Agent MuJoCo and SMAC
Offline Multi-Agent Datasets We select Multi-Agent MuJoCoode Witt et al. 2020) and the StarCraft Multi-Agent Challenge (SMAC)(Samvelyan et al. 2019) as our experiment environments. Multi-Agent MuJoCo, a benchmark for continuous multi-agent robotic control, is built on the MuJoCo environment. The Multi-Agent MuJoCo dataset we use was collected using the HAPPO(Kubx et al.) 2021) algorithm by (Wang et al. 2024) which contains four quality levels: expert, medium, medium- replay and medium-ent eeppert The second environment, SMAC, evaluating cooperative MARL methods. The offline SMAC dataset is collected by (Meng al.) 2021), using online-trained MAPPO(Kuba et al.) 2021) agents. This is the largest publicly available dataset for SMAC and includes three quality levels: good, medium, and poor. We focus three representative battle maps in our experiments: one hard map $5m_v__6m and two super hard maps (6h_ vs 8z and corridor). We initialize the behavior policy mi through behavior cloning (Michie et al. 1990 using the offline dataset. Further details on these datasets are provided in Appendix D
widely-used benchmark for
Baselines and Comparative Evaluation We compare our approach with seven offline MARL algorithms: The multi-agent versions of BCQ(Fujimoto et al., 2019) and CQL(Kostrikon al.)
2021b) (referred to as BCQ-MA and CQL-MA), ICQ((ang et al. 202I), OMAR
al. 2022).
AMIGA(Wang (Shao et al. 2024). OMIGA Wang al. 2024), and (oua et al.) 2024
Both
BCQ-MA and CQL MA utilize a linear weighted value decomposition for the multi -agltttgenttt setting.
similar to Eq. (I). Hyperparameters used in our experiments are provided in Appendix E.4 Figure
presents returns for the offline Multi-Agent MuloCo and SMAC tasks with 5 random seeds.
Detailed
analysis of the results and the mean and standard deviation of returns are in Appendix
Talus spant
Figure 3: Analysis on the influence term on Ant(left) and Hrpper(right)
8

---

Published as a conference paper at ICLR 2026
Analysis on the Influence Term In OMCDA, the influence of each agent the system is the core metric for allocating conservatism levels. We conduct experiments to analyze the relationship between the computed influence of each agent and its corresponding return. The results in Figure. demonstrate that agents with higher Vi", representing higher individual returns, tend to be allocated more influence, enabling them to have stronger impact system performance. This proportional allocation allows high-return agents to further contribute to global objectives and optimize overall behaviour. By adjusting conservatism levels properly, OMCDA enhances individual performance and maximizes collective return, promoting balanced and efficient learning across agents.
Analysis on the Components of OMCDA pact of Q-function decomposition in OMCDA, we conduct three ablation studies: OMCDA-w/o-CUD OMCDA-W/O-de and OMCDA-rd. In OMCDA-W/O-CDA, all agents share the same conservative degree d without allocation. In OMCDA OMCDA-w/o-dq, dynamic allocation remains but Q-function decom- position is removed, entangling return optimization with deviation handling. OMCDA-rU assigns each agent random d4, used evaluate the role of strategic assignment. Experiments on HalfCheetah and th_vs_8z 8z in the Multi-agent MuloCo and SMAC environments show that OMCDA consistently out- performs all ablated versions (Figure. -b.. Lacking dynamic allocation, DMCDA-W/O-CLI causes imbalance and degraded performance. OMCDA-W/o-qq weakens learning as objectives become entangled, while OMCDA-rd performs worse since random d; ignores agents' distinct impact. These results confirm that dynamic allocation and Q-function decomposition are essential for collaboration and efficiency in offline multi-agent enviromments, while strategic assignment of conservatism is crucial for optimal system performance.
To analyze conservative degree allocation and the im-
(a)
(b)
(c)
(d)
Figure 4: Analyses and ablations of OMCDA. We analyze the effect of model components (a-b) and total conservative degree (c-d) across HalfCheetah from MA-MuloCo and 6h vs 8z from SMAC.
Analysis the Total Conservative Degree The total conservative degree deot controls how much the system may deviate from the behavior policy. sets the permissible deviation for the entire system, ensuring agents do not diverge excessively. In experiments high-quality datasets, a smaller deot outperforms other settings. This is because in such environments is essential for policies to stay closer the behavior policy for stable performance. Meanwhile, dynamic allocation of d, allows agents with significant impact on returns some flexibility to deviate, while requiring others to remain conservative and adhere closely the behavior policy. The results in Figure. 4 c- show that adjusting diot improves overall performance, allowing influential agents beneficial deviations while maintaining system stability.
HalfCheetah and 6h vs_8z, based on
CONCLUSION
In conclusion, a novel offline MARL framework OMCDA is introduced to tackle the challenge of conservative degree allocation. OMCDA decomposes the Q-function in offline MARL. with regularization into two components: one for computing the return and another for capturing deviations from the behavior policy. dynamically adjusts each agent's conservative degree based on their influence the overall system's 's performance, ensuring coherent credit assignment and robust performance throughout the learning process. Meanwhile, extensive experiments demonstrate that OMCDA consistently outperforms existing offline MARL methods across various environments. Our future works aim to enhance OMCDA by developing adaptive mechanisms that reduce sensitivity to total conservative degree selection, and lower the computational complexity.

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENTS
Yun Hua is supported by the Shanghai Post-doctoral Excellence Program (2025251). Wenhao Li is supported by the NSFC (62406270) and the STCSM Shanghai Rising-Star Program (24YF2748800), Xiangfeng Wang is supported by the NSFC (62231019) and SHEITC 01004).
REFERENCES
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-dree driven offline reinforcement learning. arXiv preprint arXiv:2202.11566, 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: urXiv:1912.00 (eee//9112.6600 2019.
Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In TARK, volume 96, pp. 195-210. Citeseer, 1996.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933 09:49933-446,,, 2021.
The Viet Bui, Thanh Hong Nguyen, and Tien Mai. Comadice: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization, 2024. URL https
7/arxiv. org/ abs/ 22100.410000 01954
Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9(1): 427-438, 2012.
Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, et al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36, 2024.
Wei-Yu Yu Chiu, Gary G Yen, and Teng- Kuei Juan. Minimum manhattan distance approach to mul- tiple criteria decision making in multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 20(6):972-985, 2016.
Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06700 19, 2020.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-201 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Pablo Hernandez-Leal Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Mulri-Agent Systems, 33(6):750-797, 2019.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi multi-agent reinforcement learning. In ECAI, pp. 1148-1155 1555, 2023.
10

---

Published as a conference paper at ICLR 2026
Woojun Kim and Youngchul Sung. An adaptive entropy- regularztaulaiiiaation framework for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. PMLR. 2023.
Iyy Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Hya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 10.06:2 2021b.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94 2016.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109. 1251, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-1199,, 2020.
Byungjun Lee, Jongmin Lee, Peter Vrancx, Dongho Kim, and Kee-Eung Kim. Batch reinforcement learning with hyperparameter gradients. In International Conference Machine Learning, pp. 9725-5735. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement leamiigg: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.0/6: 006644 2020.
Wenhao Li, Xiangfeng Wang, Bo Jin, Jingyi Lu, and Hongyuan Zha. Learning roles with emergent social value orientations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu, Yile Liang, Donghui Li, and Xuetao Ding. Offline multi-agent reinforcement learning via in-sample sequential policy optimization, 2024. URL
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Picter Abbeel, and Igor Mordatch. Mueti-agent actor-critic for mixed cooperative-compettiie environments. Advances in neural information processing systems, 30, 2017.
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. arXiv preprint arXiv:21 22.22222.22255... 2021.
Donald Michie, Michael Bain, and J Hayes-Miches. Cognitive models from subcognitive skills. IEE control engineering series, IEE control engineering series, Jan 1990.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
James OrT and Ayan Dutta. Multi-agent deep reinforcement learning for multi-robot applications: A survey. Sensors, 23(7):3625, 2023.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International conference on machine learning, pp. 17221-17237. PMLR, 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regessedeeeeeddddeseeeerio Simple and scalable off-policy reinforcement learning. arXiv preprint 190.00711 2019.
Fernando Pérez-Cruz Kullback-leibles divergence estimation of continuous distributions. In 2008
IEEE international symposium information theory, pp. 1666-1670. IEFF, 2008.
11

---

Published as a conference paper at ICLR 2026
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson, Monotonic value function factorisation for deep aulti--gentt reinforcement learning. Journal of Machine Learning Research, 21 21(178):1-51, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 17Xiv:1902.04043 2019.
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conserva- tive q learning for offline multi-agent reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Valoo-decompoosiiion networks for cooperative multi-agent learning. arXiv preprint arXiv: arXiv: 7706.5526, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-leaming. arXiv preprint arXiv:2008.0100 010622 2020a.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08003, 2020b.
Xiangsen Wang and Xianyuan Zhan. Offline multi-agent reinforcement learning with coupled value factorization. arXiv preprint arXiv:2306.0890 08900, 2023.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems, 36, 2024.
Ziyan Wang, Yali Du, Yudi Zhang, Meng Fang, and Biwei Huang. Macca: Offline multi-agent reinforcement learning with causal credit assignment. arXiv preprint arXiv:2312.03644, 2023.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offiine reinforcement learning.
arXiv preprint arXiv:1911.11366 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08 2021.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:21 10.00395, 2021.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXhv:2303.15811 2023.
Yigin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 14:10299-10312, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 13:141129-2 144119944442 2020.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. arXiv preprint arXiv:2105.07333 2021.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International conference on machine learning, pp. 12491-12500. PMLR, 2021.
Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models, 2025. URL
12

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS (LLMS)
We employed the large language model as auxiliary tool during manuscript preparation. Specif- ically, was used to refine language for grammar and clarity, and to generate illustrative (non- experimental) figures based on prompts we provided. All research ideas, methods, experiments, analyses, and conclusions were developed by the authors.
B RELATED WORK
B.1 OFFLINE REINFORCEMENT LEARNING
Offline reinforcement learning must address distributional shift (Kumar et al.) 2019). which occurs
when policies encounter -iutt--diitiittti (OOD) states or actions (Fujimoto et al. 2019), leading to exploitation errors and poor performance due to inaccurate value estimates OOD actions.
To mitigate this, policy constraint methods (Cheng et al.) aim to keep the learned policy close to the behavior policy, reducing deviations from the training data. Value regularization techniques (Kostrikov et al. 2021a) (Kumar et al., 2020) penalize OOD) value estimates, while uncertainty-bast
2024 Fujimoto et al. 2019 Xu et al. 2021)
(Bai al.) 2022 Wu al. 2021) and model (Yu ett al. 2020 ZZhn et al., 2021) approaches focus on penalizing actions in uncertain or sparse regions.
Recently, in-sample learning methods (Brandfonbrener et al.) 2021 Kostrikov et al.) 2021 Peng et al. 2019 Xu et al., (023) have focused on learning within the support of the offline data, avoiding OOD evaluation and improving stability. Our approach integrates aulti-agent value decomposition into this paradigm, ensuring more stable and coordinated policy learning in multi-agent settings.
B.2 MULTI-AGENT REINFORCEMENT LEARNING
A key challenge in MARL is the joint action space (Hernandez-Leal al. 2019): which grows exponentially with the number of agents, making difficult to find optimal policies. The Centralized Training with Decentralized Execution (CTDE) framework (Kraemer & Banerjee 2016)] Oliehoek ctt al. 2008 Sunchag et al. 2017) addresses this by training agents centrally with global information, while they execute based on decentralized policies using only local observations.
Recent offline MARL approaches (Jiang & Lu 2023
Pan et al., 2022 Shao et al. 2024 Wang
al., 2024 2023 Yang et al., 2021] Zhu et 2025 Liu al. 2024 Bui et al. 2024). extend
online MARL methods with regularizee regularization to avoid OOD actions. For instance, ICQ (Yang et al.
2021) uses importance sampling for local policy constraints, while OMAR (Pan et al. 2022) adapts conservative Q-learning. In contrast to value decomposition methods, which adhere to the IGM principle, ComaDice (Bui et al., 2024) employs stationary distribution shift regularization to combat the distribution shift issue. MADiff (Zhu et al.)
2025) uses attention-based diffusion model to effectively model agent collaboration. InSPO (Liu et al. 2024) sequentially optimizes agent policies in in-sample manner. MACCA (Wang et al.) 2023) and OMIGA (Wang al. 2024) introduce global-to- lo-call value regularization. However, these methods apply a fixed conservatism level for each agent, which be inefficient. Although CFCQL (Shao et al. (0224) incorporates conservative value estimation, it fails to account for the heterogeneous impact of individual agents on overall system performance. Our algorithm addresses the above problems by dynamically adjusting conservative degree based on each agent's 's impact on the system.
C PROOFS
Proposition 3.1 In an offline MARL setting, the optimal global policy Tioe 0) is given by Eq. and is formally expressed as follows:
Tio (a o) To(a () exp (((
(26)
where Q" (o, a) is optimal action value function, (o) is the optimal value function, and we assume there is global conservatism level a that controls the overall deviation from the behavior policy.
13

---

Published as a conference paper at ICLR 2026
Proof. The proof follows Yang al. 2021) and is the extension of SAC Haarnoja al. 2018 into offline multi-agent setting.
Let us return to the definition of offline MARL with regularization, start with the original form:
max Ea 1Qtat a)l,
(27)
S.t. (TIo Tb) S, E. Tene
We find that the objective is linear function of the decision variables Teot and all constraints are convex functions. Thus Eq. (27) is convex optimization problem. The Lagrangian equation is:
ClTeot A) =EanTioe (Qtor(o, a) + a (c (Tio Tb))
Σ Teoe 1))
(28)
+ A
1 -
where a denotes the Lagrangian coefficient which a global conservatism level that controls the overall deviation from the behavior policy. Then we can get the following formula:
0C
Qroc(o, a) (1+
Aeoo(a |0)
Ta 0) )) A.
(29)
iTot
ac Setting Usiet
to zero, then:
a)) - (01111100 (1+
Teoc(a o)
))
O,
(30)
Tola 0)
Qtoc(o, =a 10(1+10 1
=a(1+l0g
Tiot 0)
To(a 0) ) + A.
(31)
Qu
x 1
Teor
log
(32)
a
a
To 0)
Qtor a
1
Teo o) TT(a 0) exp
(33)
The optimal policy expressed similar to Eq (3) while adding optimal symbol all functions, which is π to n². Integrating (3) with optimal symbol into the expression of optimal V-function in offline MARL with regularization, we can get:
Σ Time |alo) o) (i) alog
Toot (a ())
To(a 0) ))
Viat (0)
(34)
Tio aa)
+a
Through (3. Tioe (a 0):
with optimal symbol and (3),, we can finally obtain the optimal global policy
The (a (0) To(a o) exp -1)0))))))0))))
(35)
Theorem, 3.3 Given Eq. 23 the optimal policy for each agent is derived, and consistency between the local optimal policies ग. and the global optimal policy Tiat is guaranteed. This consistency holds for individual ag assignments across agents.
14

---

Published as a conference paper at ICLR 2026
Proof. To provide the proof, initially return to the decomposition framework of the Q-function in MARL setting, which is:
Q Vio (o, + a (o,a).
(36)
In this decomposition framework, the global Q is divided into two parts: Viot represents the computation of the return, and Q serves the global mapping of each agent's conservatism level.
Consider global perspective that global Ottor controls the whole conservatism level:
Q a) Viot (o,a) + Csso Qiot (o,a).
(37)
Compare Eq. (36 with Eq.37 the computation of the return is the same, while the deviation part varies due to the conservatism level. These two equations implicitly indicate that:
N
Ostt Via (0,
Σ
as Q (o.
(38)
Back to the definition of Qoi and Qioe According Eq. we have:
Zioe 0) (a 0)
Σ a log
xt (a; o:) Tb(a; o.)
Ottot loog
(39)
Then we separate the parts involving Q-function and V-function from the parts involving æ Eq.
26):
La 10)
(_ (Q"(o.a)-V((())
1
exp
(40)
aa ())
asot
aioe(a 0)
log
as(a 0)
-Q"().a)-VV()))
(41)
asot
Tine (a 0) -'(( as(a 0)
Osot -log
(42)
Similarly, the local parts in Eq. 23) can be written as
n (a: 0s) no(a, o.) log
(o) (QF (o). - V. (o:)
(43)
+ (Q (o.a) Vc,t" (o)
With Eq.(13
(16. we can sum both sides of Eq. 43 with respect
Σ a; log
n (a=
0i)
Σ (o) (Qr' (oi, az) - V,"
-V;" (ai)
To(a; oi)
(44)
Σ Σa a: (Qa (o, -oe. vo)(o))
Tiot (a o)
1a (o) (or (or, V³ vo(()))
Atot log
(a 0)
(45)
+ Σ as (Q== (o. - voii vo((o))
T ()
Atot -log
= Q" a)) - V-)
(46)
Ts(a 0)
15

---

Published as a conference paper at ICLR 2026
The transformation from Eq 44 to Eq. (66) implies that with Eq. 23
Tiot
0)
II n (a; o;),
which means (5) not only allows for the derivation of the optimal policy for each agent, but also ensures consistency between the local optimal policies मद and the global optimal policy Thots even when each agent has distinct oq.
L
Proposition 3.4 From Eq. (24). each agent's conservatism- value function V is updated through the following optimization:
(o( (Q; (0,, az) V? (o:))
min E (oLa))D
exp
(47)
(Q (0, yod -oo)) + w (o)V (0:) + a Vc.
Proof. The proof (similar to (Wang et al. 2024)) follows by showing that the first-order optimal condition of the above optimization objective, where the derivative with respect to Voi equals zero, is exactly the Eq. (2):
(o) (Q7
exp
@v)i(()
(48)
+ (Q - (o + w (o) (a)) + a Vo. (o)
()
Bar
exp
(Q) 0z) - (o:))
O
(49)
+ (Q, - vo)()))) + = 0.
From the perspective of seeking the optimal function, we can have:
w (0) (Qr (o,, - VT -VT
exp
(50)
(o.) vef (o)) -o
=1
This result implies that the optimal form of Vi" can be obtained by solving the convex optimization problem in Eq. 47
D EXPERIMENT SETTINGS
We select Multi-Agent MuJoColde Witt et al. 2020 and the StarCraft Multi-Agent Challenge (SMAC)SSamvelyan et al. 2019) experimental environments.
Multi-Agent MuJoCo, a benchmark for continuous multi- robotic control, is built on the MuJoCo environment. The Multi-Ages MuJoCo dataset use collected using the HAPPO(Kuba et al.
16

---

Published as a conference paper at ICLR 2026
2021) algorithm by (Wang et al. 2024) which contains four quality levels: expert, medium, medium- replay and medium-expert. The expert dataset is generated by employing the converged HAPPO algorithm, which involves training the algorithm until it reaches state of convergence, where the agents have learned optimal policies. The medium dataset is generated by first training policy online using HAPPO, early-stopping the training, and collecting samples from this partially-trained policy. The medium-replay dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the medium level of performance. The medium-expert dataset is constructed by mixing equal amounts of expert demonstrations and suboptimal data. For all datasets, the hyperparameter env gow__aggs.gee aas_aggnnnntsttttt_e obsk is set 1. The average returns of the datasets are listed in Table 2
Table 2: The multi-agent MuJoCo datasets.
Scenario
Quality
Average Return
2-Agent Ant
expert
2055.07
2-Agent Ant
medium
1418.70
2-Agent Ant
medium-expert
1736.88
2-Agent Ant
medium-replay
1029.51
3-Agent Hopper
expert
2452.02
3-Agent Hopper
medium
723.57
3-Agent Hopper
medium-expert
1190.61
3-Agent Hopper
medium-replay
746.42
6-Agent HalfCheetah
expert
2785.10
6-Agent HalfCheetah
medium
1425.66
A-Agent HalfCheetah
medium-expert
2105.38
A-Aeent HaliChectah
medium-replay
655.76
The second environment, SMAC, is widely-used benchmark for evaluating cooperative MARL methods. SMAC consists of set of StarCraft II micro scenarios, and all scenarios confrontations between two groups of units. Agents based on the MARL algorithm control the first group's 's units, while built-in heuristic game AI bot with different difficulties controls the second group's 's units. Scenarios vary in terms of the initial location, number and type of units, and elevated or impassable terrain. The available actions for each agent include operation, move[direction) attack [enemy id], and stop. The reward that each agent receives is the same. The hit-point damage dealt and received determines the agents' share of the reward. The offline SMAC dataset is collected by (Meng et al.) 2021), using online-trained MAPPO(Kuba et al.) [02[] agents. This is the largest publicly available dataset for SMAC and includes three quality levels: good, medium, and poor. We focus on three representative battle maps in our experiments: one hard map (5m_vs vs 6m) and two super hard maps (6h_vs_8 8z and corridor). The task types of the maps are listed in the Table For each dataset in a map, we randomly sample 1000 episodes as our dataset. The average returns of SMAC datasets are listed Table 4
Table 3: SMAC maps for experiments.
Map Name
Type homogeneous & asymmetric
5m vs 6m
6h_vs. vs 8z
micro-trick: focus fire
corridor
miero-trick: wall off
E IMPLEMENTATION DETAILS
E.1 ALGORITHM SUMMARY
In this section, we will give an explanation of the pseudocode for OMCDA. The psedudocode is shown in Algorithm. | We initialize the behavior policy म. through behavior cloning (Michie et al.)
17

---

Published as a conference paper at ICLR 2026
Table 4: The SMAC datasets.
Map Name
Quality
Average Return
5m vs 6m
good
20.00
5m vs 6m
medium
11.03
5m_ vs 6m
poor
8.50
6h vs 8z
good
17.84
6h vs 8z
medium
11.96
6h_vs_88 8z poor
9.12
corridor good 19.88
corridor medium 13.07
corridor poor 4.93
Algorithm Pseudocode of OMCDA
Input: Offline dataset D, deot Initialize return-based state-value network V/, constraint-base state-value network V;, return- based action-value network Q constraint- valuee network Q conservatism level a, and policy network тं for agent i = 1,2,...,,
for t to max-step do
Sample batch transitions (o, 0, r, d') from D.
Update return-based state-value function (o) for each agent i, via Eq. (51
Update constraint based state-value function (o) for each agent i, via Eq. 25
Compute Vi (o') and Q (o, a). via Eq. (15) and Eq. (13).
Update return- return-based action-value network O (o. a), via Eq. 53
Update constraint-be baase action-value network (? a). Eq. 52
Update local policy network for each agent i, via Eq. (54).
Calculate m, with Eq, 19) and update each agent's conservative degree de via Eq. 177
Update each agent's s conservatism level a; via Eq. (55).
end for
1990) using the offline dataset. The procedure begins by initializing all necessary networks and parameters for each agent. At each iteration, the algorithm samples transitions from the dataset D) and performs sequential updates of both local and global networks.
1. State-Value Updates: The state-value functions V and V are updated first. Inspired by (Kostiir et al., 2021b) we can implicitly update V by leveraging the expectile loss, thus avoiding the use of -fist--ditti data. V. and V. are updated following:
Update V The return-based Mate-value function V?" for each agent is updated by minimizing the following objective:
min B(o,..)> [Lg (Q, (o,, a.) Vi" (ot)).
(51)
where LÝ denotes the expectile loss with parameter т, balancing the updates based on the agent's value estimation errors.
Update V The constraint-basee
value function (o) is updated using Eq, (5)
Global Value Computation: In this step, we compute the global term for based function. Here V (o') and (o, a) are calculated in (15) and Eq. (13).
Each agent's action-value networks O and Qf are then updated. This step ensures that the agents maintain the correct mapping between their actions and the expected return as well as conservatism constraints.
Action-Value Updates:
Update Q:: The constraint- action-value function Q along with the weight w and bias b updated by minimizing the following objective, while Qad and Vea are from Eq. 14) and Eq. 16).
(Q a) 7V (o'))
min
E(,,,,0'~0
(52)
18

---

Published as a conference paper at ICLR 2026
Update Q;: The return-based action-valus function Qj, weight w/, and bias b' are updated according to the following minimization objective:
min w0.00
B( (r(o, + tyVoo (o') - Viol a))
(53)
4. Policy Updates: The agent's policy network is updated based on optimizing the following function.
Update Th; The policy T. for each agent is updated by enforcing the KKT condition Eq. 5 leveraging Eq (2)
W (Q -V
max B aa
exp
a₁
(54)
+ (0,) log m. (as o.))
Finally, each agent's conservative degree d; is updated to ensure the balance between the risk and flexibility for each agent. After calculate m, with Eq. TT), we can update d, following (1). While the conservatism level 0g is adjusted to control the balance between deviation and conservatism.
Conservatism Updates:
Update a;; The conservatism level a is updated according to 55.
min E (o,,a:)< a4 d, az DKL (T T!))
(55)
the deviation from the behavior policy is less than d,, as will decrease, allowing more flexibility for exploration. Conversely, if the deviation exceeds d,, az will increase, pushing the policy to stay closer to the behavior policy.
E.2 DETAILS OF OMCDA
The return-com computatatt and constraint modules in the Q-function and V-function, and policy networks of OMCDA are represented by 3 3-layer ReLU activated MLPs with 256 units for each hidden layer. For the both weight networks of the two modules, we use 2-layer ReLU-activated MLPs with 64 units for each hidden layer. All the networks are optimized by Adam optimizer.
For the computation of the influence term, practice, directly computing the derivatives can indeed lead to numerical instability. Therefore, we employ several techniques to stabilize the differentiation process: For continuous action spaces such as MuJoCo, we adopt a reparameterization method and use single-sample average for obtaining expectations for algorithm stability, simplifying the original differentiation process into a relationship between Q" and the log variance of policy network, which is then directly computed using deep learning libraries in PyTorch. For discrete action space environments like SMAC, due to the finite action set, we approximate the target derivative by applying small parameter perturbations to æ and using finite difference approximation trick.
In this paper, all experiments are implemented with Pytorch and executed on NVIDIA A100 GPUs.
E.3 DETAILS OF BASELINES AND COMPARATIVE EVALUATION
We compare our approach with seven recent offline MARI. algorithms: The multi-agent versions of BCQ(Fujimoto et al.) 2019)) and CQL et al. 2021b) (referred to BCQ-MA and CQL- MA), Yang et al. 2021) OMAR Pan et al.| 2022). CFCQL (Shao et al. 2024), OMIGA WWang al. 2024). and ComaDICE (Bui al. 2024). Both BCQ-MA and CQL-MA utilize linear weighted value decomposition structure for the multi-agent setting, similar to Eq. ①
Table 6 and Table 7 presents the mean and standard deviation of average returns for the offline Multi-A Agent MuJoCo and SMAC tasks with 5 random seeds. In these multi-agent scenarios, the complexity of the environment makes it challenging assign conservative degree individual agents,
19

---

Published as a conference paper at ICLR 2026
different agents' deviations from their behavior policies have varying impacts on the environment, which in turn influences the learning process. The dynamic conservative degree allocation mechanism in OMCDA assigns different conservatism levels to each agent based on their varying impacts the system, which leads to better overall system performance. Moreover, by separating the return optimization from policy deviation management, OMCDA provides resulting in improved stability and effectiveness, enabling better collaboration and more efficient policy learning compared other oflline MARL methods.
refined learning process,
Table 5: Hyper-parameter of OMCDA.
Hyperparameter
Value
OMCDA
Value network for return learning rate
2e-4
Value network for constraint learning rate
4e-5
Alpha learning rate
le-5
Policy network learning rate
2e-4
Optimizer
Adam
Target update rate
0.005
Batch size
128
Discount factor
0.99
Hidden dimension
256
Expectile parameter
0.7
Initial conservative degree d,
0.05 or 0.1 or 0.2
E.4 HYPERPARAMETERS
For multi-agent MuJoCo and SMAC, the hyperparameters of OMCDA are listed in Table Since aim to quickly learn the return while maintaining stability in deviation, we use different learning rates for the value network: one for return and another for the constraint, respectively. In OMCDA, the conservative degree d is an important parameter. When the value of is large, the algorithm's 's overall conservative degree increases, providing the system with greater flexibility in policy exploration. Conversely, when the conservative degree is smaller, the policy tends align more closely with the behavior policy. the multi-agent MuloCo environment, for the expert dataset, we set the initial d, 0.05 for each agent to guarantee effective regularization, while for other datasets, we set the initial d, = 0.2 to maintain moderate deviation. In the SMAC environment, for the good dataset, set the initial d; 0.1 for each agent, and d; 0.2 for the other datasets.
to 2 10-4 and 10-3
E.5 DETAILS OF EQ. 5
Consider common case where there's only global constraint:
max, E o Arre, s.t. Easy [oog (((a))))) salar o.))
< deot
(56)
Then according to the Lagrangian relaxation, the global conservatism level Asso can be assigned and the Q-function is formulated as:
(0. EE
Σ
r (re asot DKL Tb))
(57)
When considering Eq. (56) under the constraints specified in Eq. (5) and Proposition 3.2 the global constraint can be transformed step by step:
(at ot)
log
< d
(58)
To 0,)
20

---

Published as a conference paper at ICLR 2026
Table 6: Offline Multi-agent MuloCo Tasks
Maltigger
2
MEXKLEY
Hagper
sedian
41.58-20002
401.37211998 88
901-79:14400
21 90
150 75
1189 361554339
K02 62666663
164
Happer
eveellr
2539-26-04
31.77215
196-99163361
3.003922
567554545545
7C4 10418-494-27
906--2421
17-9-77-379771
Rogger
3516
73T
107
5500ME 79644
1406366:122 123.99
141844855
14:5
Are
19077
18
TOR 1T.1
201 BATTN
IT0E39 75.1S
1390 15A1
VIE-Y 18NTS
TRRK 21.27
Halfchestal
oueen
1902
2395
mehim
100594
308300
HalChheoa
706-70089
2838/53-92440
2948
3859.7881.
4883776-36671
log Ti(a of)
Eann
S Σ di,
(59)
To(ai o;)
Tr. (a of) mb(a) of) S di, = = Vi I.
Eavxx
log
(60)
Therefore, we effectively achieve equivalent transformation from global to local policy constraints. By comparing the Q-function under the global constraint Eq. (57) with that under local constraints in Eq. (6), and noting that Ensa log [[(a]lo() Tota]ooj] Easwiss log mo(a;))))
the following conclusion
can be derived from this equivalence:
@ 0)
(Te Oxsot DKL || [b))
E
Tt Eai DKL(T) ll mb)
(61)
x(a ( Tb(aa o)
Σ as log
nte (ai o;) Tb (a: 0:)
Oltot
log
(62)
Eq. 62 effectively decomposes the global deviation term into local components, thereby establishing the foundation for both generating Eq. (38) and the deviation term Eq.
12
Table 7: Offline SMAC Tasks
SMAC
Taok
bataeet
BOQ-MA
COL-MA
100
OMAR 7.0
CPCOL
OMIGA
ComatICE
OMCIA
5m fim
776-0011
K06-8221
7,7+0.90
8.19H0
875-6637
$7-05
18,47-0224
73820.10
011
TASEOST
35510035
75221037
$7.04
1819IWIS
pas
761:0.36
145:0100
427
7.5230.1
good
1522-1211
522-081
15.411112
1530:0.80
14:73:0000
1082-092
1266071
44
1166-1.30
1306-171
cornikor
poor
mediam
&r
poor
1084-0.16
10.1+0052
10.55+0.10
106744025
10:30-0.47
1131-0.19
11.4006
10.81+0.10
F ADDITIONAL RESULTS
F.1 ANALYSIS ON THE COMPONENTS OF OMCDA
To analyze the solution to conservative degree allocation and assess the impact of Q-function decompo- sition in OMCDA, we conduct three distinct ablation studies: OMCDA-W/-CDA OMCDA-w/o-dg and OMCDA-rd. In MMCCDA-W/-CDDA. all agents are assigned the same conservative degree d,, without the implementation of conservative degree allocation. In contrast, OMCDA-w/o-de maintains the dynamic conservative degree allocation but eliminates the Q-function decomposition, preventing the separation of return optimization from policy deviation handling. OMCDA-rd introduces random allocation of the conservatism constraint, assigning each agent a random d, in which we hope to evaluate the importance of strategically assigning conservatism levels based each agent's 's impact.
Experiments are conducted on the HalfCheetah and 6h vs 8z tasks in the Multi-agent MuJoCo and SMAC environments, respectively. Figure. shows that OMCDA consistently outperforms all
21

---

Published as a conference paper at ICLR 2026
5003
4500
14-
4003
13
3509
rvura réurn
A
1009
11
2009
10
IMCDAA-vi-CIDA
OMCDA-wIl-CCDA
1000
RMCCA--wbbdd
OMCDA-w wioDd
OMCDA IMCDA-
OMCIDA-woord
0.0
0.2
0.4
60
108
limesteprimillion)
Ephsodesthoussan
(a) HalfCheetah-eppert
(b) 6h__s_ _h_______ fs_8z good
Figure 5: Analysis on components of OMCDA
ablated versions across both tasks. In OMCDA-w/O-CDA, the absence of dynamic conservative degree allocation results imbalanced agent behavior, with some agents being overly conservative and others overly aggressive, leading to performance degradation. OMCDA-w/O-oe exhibits weaker results due to the entanglement of return maximization and constraint handling, which complicates learning and produces suboptimal policies. OMCDA-rd, which applies random conservatism allocation, demonstrates inferior performance, as randomly assigned d, values fail to account for each agent's unique influence system performance.
These results confirm that both the dynamic conservative degree allocation and Q-function decompo- sition essential for achieving better collaboration and learning efficiency in offline multi-agen environments, while the strategic assignment of conservatism is crucial for optimizing system perfor-
mance.
F.2 ANALYSIS ON THE TOTAL CONSERVATIVE DEGREE
4500
4000
13
Avurag reture
rdura
3000
12
1500
2000
1509
18
d 1ot-0.3
1ot-0.5
1000
d 1ot-1.2
d bot-
500
4 1ot-3
0.0
0.2
8.4
0.6
0.8
1.
10
20
30
40
50
60
70
80
90
100
limestepsttilllinn)
Eplsndesithossane
(a) HallCheetab-exppet
(b) vs good
Figure 6: Analysis on the total conservative degree
The total conservative degree dest controls how much the system is permitted deviate from the behavior policy. It establishes the total permissible deviation for the entire system, ensuring that agents do not diverge excessively from the behavior policy, helping to avoid the introduction of suboptimal actions into the system.
In our experiments on the HalfCheetah and 6h, vs 8z environments, which are based on high-quality datasets, smaller diat outperforms the other settings. This is because, in such environments, it is essential for the policies to stay closer to the behavior policy for stable performance. At the same time, the dynamic allocation of d, allows agents that have significant impact on the system's return
22

---

Published as a conference paper at ICLR 2026
to have flexibility to deviate, while requiring other agents remain more conservative and adhere closely to the behavior policy. The experimental results in Figure. ddemonstrate that properly adjusting deot improves system performance and allows influential agents to achieve beneficial deviations while maintaining overall system stability.
Values agent
Values agont
Válues agent
Infuueee for
Infruance for agent
Influnnee for agent
Valoe
Timeseps(Million)
Figure 7: Analysis on the Influence Term on Hopper-medium-reple.
Values for agent
Values for agent
Influmec agent
Infiuesce agent
12
Vilue
10
8.5
Inffuence
6/2
Timesteps(Millloon
Figure 8: Analysis on the Influence Term on Ant-medium-expert
23

---

Published as a conference paper at ICLR 2026
F.3 ANALYSIS ON THE INFLUENCE TERM
In OMCDA, the influence of each agent on the system is the core metric for allocating conservatism levels. To determine whether have accurately assessed each agent's influence in the environment, we conduct experiments to analyze the relationship between the computed influence of each agent and its corresponding return.
The results in Figure. 7 and Figure. 8 demonstrate that agents with higher V.", representing higher individual returns, tend to be allocated more influence within the system, enabling them to have stronger impact on system- wide performance. This proportional allocation allows high-return agents further contribute global objectives and optimize overall system behaviour.
By adjusting conservatism levels properly, OMCDA enhances individual agent performance and maximizes collective system return, promoting balanced and efficient learning across all agents.
24


Reviews:
[{"review_id": "q7ikOWUXKF", "reviewer": "Reviewer_mivS", "summary": "The authors introduce Offline MARL with Conservative Degree Allocation (OMCDA) which aims to control the degree of conservatism for each agent by decomposing the Q-values to return maximizing and conservatism terms. They further use value decomposition akin to QMIX to compute the degree of conservatism for each agent based on how the KL regularization term impacts the global $V$. Experimental results are provided for SMACv1 and MA-MuJoCo.", "questions": "1. While the illustrative example in Table 1 makes some sense, what happens if the payoffs for taking action A for agent 1 are [-100, 3 ] instead of [3, 3]? \n2. CFCQL and OMIGA seem to be important baselines since they both consider a similar problem of applying individual conservatism. How did you tune their hyperparameters? Is it similarly tuned fairly for OMCDA?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The  illustrative matrix game is somewhat intuitive. Under some fixed total conservative degree, agent A has a higher influence on total reward, so it should follow more risky behavior, and deviate from the dataset distribution (assuming the higher return action is also in the dataset support). \n\nAlso, updating $\\alpha$ in Eq.55 is interesting since we often want less hyperparameter tuning in offline RL.", "weaknesses": "1. The observation function $O$ is missing from the Markov Game formulation. \n1. The reward function $r$ is defined in terms of joint observations, but it should be based on state. \n1. The example used to motivate the work is slightly confusing. For football, the risk-taking vs risk-averse behavior between positions is about taking actions that can lead to states with a high return (i.e. score a goal) but may have low probability of reaching that goal. This is somewhat orthogonal to the concept of conservatism in offline MARL, where risk-averse vs risk-taking is about whether to follow the dataset distribution or not.\n1. $Q_i^r, Q_i^c,  w_i^r, w_i^c$ from Eq. 13 and 14 is not well motivated. It seems that additional decomposition assumptions are required on both the reward component and constraint component. There is no detail provided in appendix E.5 as the authors mention. \n1. Related to above, the validity of Eq.23 is questionable. For instance $Q^{c, i, *}$ is not well-defined. If value decomposition is used, the authors should be more clear about how the IGM assumption is extended to the offline setting and how it is affected by the conservative regularization term.\n1. Even with a correct characterization of the decomposed terms, value decomposition is already quite restrictive. For instance, [1]  showed that it cannot solve simple matrix games, and [2] analyzed the limitations of value decomposition in the offline case. \n1. Eq. 18 computes the influence term depending on how the KL term influences return. However, there is still a mismatch because the denominator is in terms of individual actions while the numerator is defined for joint observations/actions. Thus, there is an implicit dependency with the other agents observations. This kind of influence term can be biased in more complex environments if there is no 1-1 mapping between individual and joint observations. \n1. SMAC results are only provided for v1, but this has already been show to be an outdated benchmark due to the deterministic transitions, and the fact that it can be solved by some simple algorithms that don’t even take observations into consideration [3]. Recent offline MARL algorithms such as ComaDICE already use smacv2 as a benchmark. This significantly weakens the experimental results of this paper. Also, smacv2 results would help with addressing my previous point since it is inherently more partially observable. \n\n\n[1] Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning (Fu et, al. 2021)\n\n[2] AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation (Matsunaga \n\n\n[3] SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning (Ellis et, al. NeurIPS 2023 Datasets and Benchmarks)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYzWi2unZH", "reviewer": "Reviewer_U5BW", "summary": "This paper introduces OMCDA, an offline multi-agent reinforcement learning (MARL) framework that dynamically allocates conservative degrees to individual agents based on their estimated influence on system returns. The key innovation lies in disentangling the Q-function into return and deviation components, enabling agent-specific regularization. The proposed method is evaluated on Multi-Agent MuJoCo and SMAC benchmarks, showing consistent performance gains over existing offline MARL baselines.", "questions": "1. What is the computational overhead of estimating influence terms compared to standard offline MARL methods, especially for N > 10 agents?\n\n2. At present the total conservative budget $d_{tot}$ is treated as a hyper-parameter and tuned manually for each task. Have the authors explored any data-driven or adaptive procedure to automatically infer $d_{tot}$ so that the algorithm can adjust the global deviation allowance without human intervention?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel Problem Formulation: Identifies and addresses the underexplored issue of heterogeneous conservatism in offline MARL — a real and important problem.\n\nTechnical Innovation: The Q-function decomposition and influence-based conservative allocation are technically sound and well-motivated.\nStrong Empirical Results: Outperforms 7 strong baselines across diverse tasks and dataset qualities, with thorough ablations validating each component.\n\nTheoretical Justification: Provides derivations for global-to-local policy consistency under agent-specific conservatism levels, enhancing credibility.", "weaknesses": "Scalability Concerns: The influence term requires computing partial derivatives w.r.t. per-agent KL divergences, which may become prohibitive in large-N systems or high-dimensional action spaces.\n\nSensitivity to Total Budget: Performance is sensitive to the choice of total conservative degree $d_{tot}$, yet no principled method is provided for setting it across tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TbHOutfUiA", "reviewer": "Reviewer_PtX1", "summary": "The paper addresses challenges in Offline Multi-Agent Reinforcement Learning (MARL), particularly the need for effective collaboration among heterogeneous agents using fixed datasets. It introduces a novel framework called OMCDA (Offline MARL with Conservative Degree Allocation) that dynamically adjusts the conservatism level for each agent based on their impact on overall system performance.", "questions": "1. For each agent's Q and conservative term, can a similar effect be achieved by mixing them through the same mixer network？\n  2. Is $\\omega_i^r(o)$ in Eq. 13, $\\omega_j^{c,i}(o)$ in Eq. 14 strictly greater than 0?\n  3. Is it feasible to use another mixer network to fit the computation of $m$ in Eq. 20?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of dynamically adjusting conservatism levels based on agent impact is a significant advancement in the field, addressing limitations of uniform approaches\n2. The paper provides extensive experimental results, showcasing the effectiveness of OMCDA across various scenarios.\n3. The paper is well-organized, with a logical flow from introduction to methodology, experiments, and conclusions.", "weaknesses": "The design of the method is overly complicated, and the rationale behind some aspects of the design has not been justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)