MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
Authors: Ziyan Wang
Yali Du
Yudi Zhang
Meng Fang
Biwei Huang
Venue: Accepted by TMLR ()

Abstract: Offline Multi-agent Reinforcement Learning (MARL) is valuable in scenarios where online interaction is impractical or risky. While independent learning in MARL offers flexibility and scalability, accurately assigning credit to individual agents in offline settings poses challenges because interactions with an environment are prohibited. In this paper, we propose a new framework, namely \textbf{M}ulti-\textbf{A}gent \textbf{C}ausal \textbf{C}redit \textbf{A}ssignment (\textbf{MACCA}), to address credit assignment in the offline MARL setting. Our approach, MACCA, characterizing the generative process as a Dynamic Bayesian Network, captures relationships between environmental variables, states, actions, and rewards. Estimating this model on offline data, MACCA can learn each agent's contribution by analyzing the causal relationship of their individual rewards, ensuring accurate and interpretable credit assignment. Additionally, the modularity of our approach allows it to integrate with various offline MARL methods seamlessly. Theoretically, we proved that under the setting of the offline dataset, the underlying causal structure and the function for generating the individual rewards of agents are identifiable, which laid the foundation for the correctness of our modeling. In our experiments, we demonstrate that MACCA not only outperforms state-of-the-art methods but also enhances performance when integrated with other backbones.

Meng Fang Published in Transactions on Machine Learning Research (06/2025)
MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
Ziyan Wang King's is College London
ziyon, siyon. www.nwwwnakkkl..cccom
Yali Du
yoli.dukkcl..co uk
King's 's College London
Yudi Zhang
y.shang5@tue.rm ml
Findhoven University of Technology
meng. wwwg..fang@llveerooooooom uk
University of Liverpool
Biwei Huang
bih007/wucsd edu
University of Colifornia San Diego
Reviewed on OpenReview: https: openreview. net/ forum? id= guU0=I4DuV
Abstract
Offline Multi-agent Reinforcement Learning (MARL) is valuable in scenarios where online interaction is impractical or risky. While independent learning MARL offers flexibility and scalability, accurately assigning credit to individual agents in offline settings poses challenges because interactions with an environment are prohibited. In this paper, we propose new framework, namely Multi-Agen Causal Credit Assignment (MACCA), to address credit assignment in the offline MARL setting. Our approach, MACCA, characterizing the generative process environmental variables, states, actions, and rewards. Estimating this model on offline data, MACCA can learn each agent's contribution by analyzing the causal relationship of their individual rewards, ensuring accurate and interpretable credit assignment. Additionally, the modularity of our approach allows to integrate with various offline MARL methods seamlessly. Theoretically, we proved that under the selting of the oflline dataset, the underlying causal structure and the function for generating the individual rewards of agents are identifiable, which laid the foundation for the correctness of our modeling. In our experiments, we demonstrate that MACCA not only outperforms -tate-of-the-art methods but also enhances performance when integrated with other backbones.
Dynamic Bayesian Network, captures relationships between
1 Introduction
Offline Reinforcement learning (RL) has gained significant popularity in recent years. It can be particularly valuable in situations where online interaction impractical or infeasible, such as the high cost of data collection or the potential danger involved (Levine al.) 2020)). In the multi-agent setting, offline multi- agent reinforcement learning (MARL) has identified and addressed some of the challenges inherited from offline single-agent RL, such as distributional shift and partial observability (Du et al. 2023). For example, ICQ (Yang et al. 2021 focuses on the vulnerability of multi-agent systems to extrapolation errors, and CQL (Kumar el al. 2020 aims to miligate overestimationn in Q-values, which can lead to suboptimal policy learning. The independent learning paradigm in MARL is appealing due to its flexibility and scalability, making a promising approach to solving complex problems dynamic environments. While independent learning in MARL has its merits, it will significantly hinder algorithm efficiency when the offline dataset only includes team rewards. This presents credit assignment problem, aiming assign credit to individual

---

Published in Transactions on Machine Learning Research (06/2025)
agents within partial observability and emergent behavior. While plain reward regression methods directly map joint state-action pairs to single scalar reward, they inherently lack interpretability, as they do not reveal which specific dimensions of states or actions drive each agent's 's individual contributions. In contrast, employing a causal model explicitly captures these critical dependencics, providing clarity on cach agent's 's causal influence and enabling more effective and interpretable credit assignment.
In offline MART, addressing the issue of credit assignment is challenging. Agents are reliant on static, pre-collected datasets, often spanning a variety of behavior policies and actions across different time periods. This diversity in data distributions increases the difficulty of assigning credits, given that the nuances of agent contributions are lost. in the plethora of policies. Recent credit assignment methods, such as SQDDPG (Wang ct al. 2020) and SHAQ (Wang et al.) 2022a), are primarily conceived for online scenarios where continuous feedback aids in refining credit assignments. However, when restricted to static offline data in oflline MARL, they miss out on the essential dynamism and agility needed to accurately understand the intricate interplay within the dataset. Morcover, in offline settings, methods like SHAQ. which rely on the Shapley value, and SQDDPG, which employs a Shapley-like approach for individual Q-value estimation, face inherent challenges. Computing the Shapley value or its approximations demands consideration of every potential agent coalition, process that is computationally intensive. In offline MARL, such approximations can lead to imprecise credit assigmments due to loss in precision, potential data inconsistencies from the static nature of past interactions, and scalability issues, especially when numerous agents operate in intricate environments.
In this paper, we propose a new framework, namely Multi-Agent Causal Credit Assignment (MACCA), to address credit assignment in an offline MARL setting. MACCA equates the importance of the credit assignment and how the agent makes the contribution by causal modeling. MACCA first mod- els the generation of individual rewards and team reward from the causal perspective, and construct a graphical representation, as shown in Figure [I over the involved environment variables, including all the dimensions of states and actions of all agents, the individual rewards and the team rewards. Our method treats team reward as the causal effect of all the individual rewards and provides a way to recover the underlying parametric model, supported by the theoretical evidence of identifiability. In this way, MACCA offers the ability to distinguish the credit of each agent and gain insights into how their states and actions contribute to the individual rewards and further to the team reward. This is achieved through a learned parameterized generative model that de- composes the team reward into individual rewards. The causal structure within the generative process further enhances our understanding by providing in- sights into the specific contributions of cach agent. With the support of thoorctical identifiability, we identify the unknown causal structure and individual reward function in such causal generative process. Additionally, our method offers clear explanation for actions and states leading to individual rewards, promoting policy optimization and invariance. This clarity enhances agent behavior comprehension and aids in refining policies. The inherent modularity of MACCA ensures its compatibility with a range of policy learning methods, positioning it as a versatile and promising MARL solution for various real-world contexts.
Ament
Amentt
Amond N
Apunti Agent
Ament Agent N
Agent 1
+)
A
al
+
R. (+)
The graphic representation of the causal structure within the MACCA framework. The nodes and edges represent the causal relationships among var- ious environmental variables, i.e., different dimensions of these variables for cach agent within the team reward Multi-agent. MDP context. These dimensions include the different dimensions of the state st..,, .u action at., ,t: individual reward rf for agent i, and the team reward R, The individual reward rf (shown with blue filling) is unobservable, and the aggregation of rī equals Re.
Figure 1:
We summarize the main contributions of this paper as follows. First, we reformulate team reward decomposition by introducing a Dynamic Bayesian Network (DBN) to describe the cansal relationship among states, actions, individual rewards, and team reward. We provide theoretical evidence of identifiabiliyy to learn the causal structure and function within the generation of individual rewards and team rewards. Second, our proposed
2

---

Published in Transactions on Machine Learning Research (06/2025)
April
Aperzh
Oftimee Dataset D
Policy Training
Anent.1
Ansoli
Anent Agent N
F2
dist1
Agent
AgentI
Agent 1
Agenti
Agent
Individual Reward Predictor
gla v
oz
aj
+
o]
o
of
$ s
t
Endronnmmts
R₁
$
a a
TT2 a
$1,11 2,1
The illustration of the MACCA method. The offline data generation process begins on the left side, where data is recorded from the environment. MACCA then constructs causal model consisting of DBN represented in grey and an individual reward predictor depicted in blue. The DBN is used to sample scales from each agent, denoted c c and highlighted in green. Meanwhile, the individual reward predictor takes the joint state, action, and these masks as imput to generate the individual reward estimate Aj. During the policy learning phase, each agent utilizes their observation and individual reward estimate as inputs, which are then passed through their respective policy network to generate the next-state actions.
Figure 2:
method can recover the parameterized underlying generative process. Lastly, the empirical results both discrete and continuous action settings, all three environments, demonstrate that MACCA outperforms current state-of-the-art methods in solving the credit assignment problem caused by team rewards.
2 Related Work
In this section, we review the close-related topics, i.e., Offline MARL and Multi-agent Credit Assignment and Causal Reinforcement Learning.
Offline MARL. Recent research (Pan et al. 2022 Kostrikov et al. 2022 Jiang & Lu 2021 efforts have
delved into offline MARL, identified and addressed some of the issues inherited from offline single-agent
RL Agarwal ct al. 2020 Yu ct al. 2020 Yang et al. 2022 Wang ct al. 2023). For instance, ICQ Yang
el al.3 2021) focuses on the vulnerability of multi-agent systems lo extrapolation errors, while MABCQ (Jing
& Lu 2021 examines the problem of mismatched transition distributions fully decentralized offline MARL. However, these studies all assume using global state and evaluate the action of the agents relying the
team rewards. Other approaches [Tseng et al. 2022) have long term progress in online fine-tuning for offline MARL training but have not taken into account the learning slowdown caused by credits of agents to the entire team. For the learning framework, the two most popular recent paradigms are Centralized Training with Decentralized Execution (CTDE) and Independent Learning (IL). Recent rescarch (de Witt et al., 2020:
Lyu el al., 2 shows the benefits decentralized paradigms, which lead to more robust. performance compared to a centralized value function.
Multi-agent Credit Assignment. Multi-agent Credit Assignment is the study to decompose the team reward to each individual agent in the cooperative multi-agent environments (Chang et al. 2003 Du al. 2019: Chen al., 2023). Recent works (Sunehag et al.) 2018 Foerster et al., 2018 Wang al. 2020
Rashid et al. 2020 Li al. 2021 focus value function decompose under online MARL manner. For instance, COMA Foerster et al. 2018) is a representative method that uses centralized critic to estimate the counterfactual advantage of an agent action, which is an on-policy algorithm. This means it requires the corresponding data distribution and samples consistent with the current policy for updates. However, in an offline setting, agents are limited to previously collected data and can't interact with the environment. This data, often from varying behavioral policies, might not align with the current policy. Therefore, COMA cannot be directly extended the offline setting without changing its on-policy features (Levine In online off-policy settings, state-of-the-ari eredit assignment algorithms such SHAQ [Wang al. 2022a
2020
3

---

Published in Transactions on Machine Learning Research (06/2025)
and SQDDPG (Wang et al. 2020: utilize an agent's approximate Shapley value for credit assignment. In the experiment section, we conduct a comparative analysis with these methods, and the results for MACCA demonstrate superior performance. Note that we focus on explicitly decomposing the team reward into individual rewards in an offiine setting under the casual structure we learned, and these decomposed rewards will be used to reconstruct the offline dataset first and further the policy learning phase. Recent work by Zhang ct al. (2024) proposes a causality-inspiredd spatial-temporal return decomposition approach for multi-agent RL, which further highlights how alternative causal structures can be leveraged to improve credit assignment. However, unlike Zhang et al., their method still assumes access to semi-online or interventional data to disentangle individual contributions, whereas MACCA operates purely on static offline data and comes with a formal identifiability guarantee.
Causal Reinforcement Learning. Plenty of work explores solving diverse RL problems with causal structures. Most research emphasizes the transferability of RI. agents. For instance, Huang et al. (2021) learn factored representations and domain-specific change factors, while Feng et al. (2022) extend these ideas to handle non-stationary environments. More recently, WWang et al. (2022b) and Pitis et al. (2022) propose removing unnecessary dependencies in causal dynamics models enhance generalization to unseen states. Hu
ct
al.
2023 exploit causal relationships between actions and reward components to reduce gradient variance
during policy learning, and Zhang et. al. 2023 leverage causal structures to solve single-agent temporal credit assignment problems. Additionally, causal modeling has been introduced into multi-agent tasks (Grimbly
let al.
2021 Jaques et al. 2019). model-based RL (Zhang & Bareinboim 2016), and imitation learning (Zhang
ct al. 2020 In the context of games, Hammond et al. (023)) extend Pearl's causal hierarchy (Pearl 2009b
to game-theoretic scenarios, developing structural causal games to analyze causal effects and counterfactual
reasoning among fully observable, rational agents. However, unlike these prior approaches, our method specifically addresses oflline settings in partially observable Dec-POMDPs, requiring a causal-identifiablllie proof to reliably recover individual rewards and causal structures from static datasets. Thus, MACCA complements existing causal RL methods by uniquely targeting causal reward decomposition to facilitate decentralized policy learning under partial observability and oflline constraints.
3 Preliminaries
In this section, we review the widely-used MARL training framework, the Decentralized Partially Observable Markov Decision Process, and briefly introduce Offline MARL.
Decentralized Partially Observable Markov Decision Process (Dec-POMDP) Oliehook al. 2016) is defined by tuple M S, P, P,R,ROO R, 7) In this tuple, N represents the number of agents, S is the state space, and A is the shared action spaces and at E A is the action for agent i. The state transition function P(s'|s, a) specifies the probability of transitioning to a new state given the current state s and joint actions a = (a' (a'.....a"), The R₂ = R(s, a) is the team reward given by the team reward function and of = O(s, i) is the local observation for agent at global state s. Each agent use a policy ( (ot) parameterized by 0 to produce an action of from the local observation o', and optimize the discounted accumulated team reward Jx = ax))), where a, (a),...,,,?) a²) is the joint action at time step t, and y represents the discount factor.
We assume that the observed team reward R2 can be expressed the sum of individual rewards, i.e. R2 = Ei' rf. Although this does not hold for every cooperative task, it is exactly true (or nearly so) in many common benchmarks (e.g. MPE, SMAC). Importantly, as we will demonstrate later via ablation studies, MACCA remains effective even when the true reward is not strictly additive.
Offline MARL. Under offline setting, we consider a MARL scenario where agents sample from a fixed datasct D= of, at, Re, sf' of'). This dataset is generated from the behavior policy me without any interaction with the environments, meaning that the dataset pre-collected offline. Here, s/. of and of represent the state, observation and action of agent at time f, while R₁ is the team reward received at time t, and sy", of' represents the next state and observation of agent i.
Dynamic Bayesian Networks (DBN) Murrhhy 2002) is a graphical model for representing the joint distribution over a sequence of random variables across time. A DBN consists of repeated "time slices," " where each slice t contains set of variables (e.g. X.), and edges within the slice encode contemporaneous
4

---

Published in Transactions on Machine Learning Research (06/2025)
dependencies. Between slices, directed edges specify how variables at time t inffuence variables at time + 1, typically under a first-order Markov assumption:
T P(X.T) = P(X.) II X1-1).
t=2
Each conditional distribution P(X, Xx-1) further factorizes according to the intra-slice structure time t. In this way, DBN compactly encodes both temporal transitions and within-slice conditional independencies..
4 Offline MARL with Causal Credit Assignment
Credit assignment plays a crucial role in facilitating the effective learning of policies offline coopcrative scenarios. In offline MARL, elfectively assigning credit among agents is inherently challenging due to partial observability and the static nature of the data. A traditional Decentralized Partially Observable Markov Decision Process (Dec-POMDP) implicitly captures temporal causality; however, it does not explicitly specify which exact dimensions of an agent's state or action affect its individual reward. A generic reward- prediction network- succhkr--chhhheee as a standard regression from joint states and actions directly to a single scalar reward-cannot reveal which agent-spocific state-action dimensions drive each agent's 's contributions, leading to limited interpretability and an absence of sparsity.
To overcome these limitations, we propose MACCA, method leveraging a causal modeling approach. Specifically, MACCA introduces Dynamic Bayesian Network (DBN) (Murphy 2002) to explicitly characterize causal relationships among variables. By learning binary masks, MACCA clearly identifies which subsets of the joint state-action variables causally influence each agent's individual reward, ensuring interpretability and sparsity. Moreover, our DBN formulation guarantees the identifiability of the true causal structure and individual reward functions under standard assumptions- manamlliiossnnaalyy the faithfulness and Markov conditions (sec Proposition4..]) These critical properties significantly enhance the effectiveness of spatial credit assignment, particularly in scenarios where individual agent rewards are unobscrvable and further interaction with the environment impossible.
In the following sections, we first present the underlying generative process within the offline MARL scenario, which serves as the foundation of our method. Then, we describe how to recover the underlying generative process and perform policy learning with the assigned individual rewards.
As shown in Figure 2 our method includes two main components: a causal model a/m and a policy model th. The overall objective, denoted as LMACCA, consists of two key terms: (1) a model estimation loss L which measures how accurately the individual reward predictor reconstructs the observed leam reward from predicted individual reward components, along with an l penalty to enforce sparsity in causal masks, and (2) offiine policy learning loss JR, computed based standard offline RL algorithms (such as CQL, OMAR, or ICQ), using the assigned individual rewards instead of the team reward. Thus, our combined loss function can be expressed as:
LMACCA = Im + Jm.
(1)
where the specific form of Jx depends on the offline RL algorithm used (e.g., JCQL JOMAR or J!CQ in this paper).
4.1 Underlying Generative Process in MARL
As foundation of our method, we introduce Dynamic Bayesian Network (DBN) (Murphy 2002) to characterize the underlying generative process. DBN is special type of graphical model that captures the temporal dependencies between variables, corresponding to state transitions across time steps in sequential decision making, By leveraging the DBN structure, we can naturally account for the graph structure over state, action, and reward variables, as well as their temporal dependencies, Icading to a natural interpretation of the explicit contribution of each dimension of state and action towards the individual rewards.
We denote the G as the DBN to represent the causal structure between the states, actions, individual rewards, and team reward as shown in Figure D which is constructed over a finite number of random variables
(s S' " d'g e rl, R2 R1) where the d' and d' correspond to the dimensions of the state and
N.1
5

---

Published in Transactions on Machine Learning Research (06/2025)
action of agent i respectively. R2 is the observed team reward at time step t. rf is the unobserved individual reward at time step L. T is the maximum episode length of the environment. Then, the underlying generative process is denoted as:
r = s(c 8t, c ciar o at i, Bix)s R₂ = 2i= 7
where Et, denotes an exogenous noise term that is independent and identically distributed across agents and time-i.e. standard structural-causal-moddd noise to account for variability beyond the deterministic mapping f. Here, 8, { - .i.....,,,,..................... d 2) and a, {ai,...., ag dg. .......,,0... ai ii, ax . are the joint state and joint action of all N agents at time i, with D. - Σi dz and D, - Ei=1 da denoting their respective total dimensions. The operator C element-wise multiplication, f is the unknown non-lineat individual reward function, and c E {0, D) cia-r E {0, 1) are binary masks (which may be dynamic or static) that indicate which dimensions of s and a, inffuence rt. In particular, if there is causal edge from the k-th dimension of 8, to agent j's individual reward r in 9, then cJ.++t(kk)= =1 We emphasize again that we assumne an additive structure for the team reward (R, -i=1 ri) to facilitate interpretable and computationally efficient credit assignment. As discussed in Section 3, this assumption typically aligns closely with standard cooperative MARL benchmarks, and importantly, MACCA remains effective even when rewards are inherently non-additive.
N
Each binary mask o; ei E (0, and cj ciarr E [0, 10, serves two purposes: (1) it reveals the causal parents of agent i's individual reward, granting interpretability; and (2) it regularizes vr so that, in an offline dataset with mixed behavior policies, 1/, does not latch onto spurious correlations among irrelevant. state-action dimensions. Without these masks, one would need to predict f (S1:N,, 0I:N,1, i) over all D. + Da dimensions making it impossible to recover "which features matter" or to prove identifiability. The masks enable both sparsity and clear DAG structure.
Proposition 4.1 (Identifiability of Causal Structure and Individual Reward Function). Suppose the joint state st, joint action as, team reward RG are observable while the individual ri for each agent are unobserved, and they are from the Dec-POMDI as described in Eq 2 Then under the Markov condition and faithfulness assumption (refer to Appendix (), given the current time step team reward Re, all the masks c- eaa-t as well as the function are identifiable.
The proposition 4.1 demonstrates that we can identify causal representations from the joint action and state, which serve as the causal parents of the individual reward function we want to fit. This allows us to determine which agent should be responsible for which dimension and thus generate the corresponding individual reward function for each agent. The objective for each agent changes to maximize the sum of individual rewards over an infinite horizon. The proof is in Appendix D
4.2 Causal Model Learning
In this section, we delve into identifying the unknown causal structure and reward function within the graph G. This is achieved using the causal structure predictor by and the individual reward predictor v/,, The set
1 = [e/g--r +iattr] is to learn the causal structure. Specifically, e/g and eha are employed to predict
the presence of edges in the masks described by Eq 2 We have
2.8- ( ar 1), 2ta w/a i);
(3)
where, ć; -: and é, aia-tr are the predicted masks for agent i at timestep t. Note that these causal masks are time-invariant and can change with state and action. We generate masks at each time step since we consider the inherent complexity of the multi-agent scenario, which has high dimensionality and the dynamic nature of the causal relationships that can evolve over time. Thus, we adopt v and vig to generate mask estimation at each time stop t, within the joint state and joint action and agent id as the input. This dynamic mask adaptation facilitates more accurate cansal modelling. To further validate this estimation, we have conducted ablation experiments Section 5.3
6

---

Published in Transactions on Machine Learning Research (06/2025)
The er is used for approximating the function f, and is constructed by stacked fully-connection layers. To recover the underlying generative process, i.e., to optimize Wr, we minimize the following objective:
I = ED Σ (r (ch caaa- St, Oo, i)]² + Lreg.
(4)
The Lreg serves as an LI regularization, akin to the purpose delineated in (Zhang & Spirtes 2011 Its primary objective is to clear redundant features during training, reduce the number of features that given depends on, and use the coefficients of other features completely set to zero, which fosters model interpretability and mitigates the risk of overfitting And it defines as:
Lreg A A1 DE |
A 42 Em ||afa-+>11> 1|1,
(5)
where Ac) are hyper-parameters. For more details, please refer to Appendix
4.3 Policy Learning with Assigned Individual Rewards.
For policy learning, we use the redistributed individual rewards Fi to replace the observed team reward RŞ- Then, we carry out the policy optimizing over the offline dataset D.
Individual Rewards Assignment. We first assign individual rewards for each agent's 's state-action-id tuple (s,, a) in the samples used for policy learning. During such an inference phase of individual rewards predictor, we first utilize hyperparameter, h, as an element-wise threshold to determine the existence of the inference phase. Elements within the mask 27.0 and ai--t will be set to 0 if their absolute value is less than h, and 1 otherwise. Then, we assign an individual reward for each agent as,
Pi ( ate cia- , cha .)).
(6)
Offline Policy Learning. The process of individual reward assignment is flexible and is able to be inserted into any policy training algorithm. We now describe three practical offiine MARL methods, MACCA-COM MACCA-OMAN and MACCA-ICQ. In all those methods, they use Q- Value to guide policy learning, for each agent who estimates the Q'(o', aa) E. EE=0 RR) with the Bellman backup operator, we then replace the team reward by learned individual reward F as (o at) Er Pil, then in the policy improvement step, MACCA-CQL trains actors by minimizing:
JOL = -)) &Bblloo Σ exp(Q'((', - Ea'nay (o',''))))))
(7)
where, y² Pl + min Q** =O0")) from Fujimoto al. (018 minimize the temporal difference error, Q' represents the target Q for the agent i, a is the regularization coefficient, ág is the empirical behavior policy of agent in the dataset. Similarly, MACCA-OMAR updates actors by minimizing:
J JOMAR E. (o) (( -a,)²).
where à; is the action provided by the zeroth-order optimizer and T e [0, 1] denotes the coefficient. For the MACCA-ICQ, it updates actors by minimizing:
JucQ Eo(LE(((ss Q'o - (o',a'))'
where L5 is the squared loss based on expectile regression and the 7 is the discount factor, which determines the present value future rewards. As MACCA uses individual reward to replace the team reward, we do not dircctly decompose value function, unlike the prior offline MARL methods (Focrster al. 2018 Wang et al., 2020 2022a),, thus we do not require fitting an additional advantage value or Q-value estimator, simplifying method.
7

---

Published in Transactions on Machine Learning Research (06/2025)
Table Average Normalized Score of MPE task with Team Reward
I-CQL
OMAR
MA-ICQ
MACCA-CQL MACCA-OMAR
MACCA-ICQ
Exp( CN)
33.6 = 22.9
44.7 = 46.6
45.0 = 23.1
85.4 + 8.1
111.7 = 4.3
90.4 = 5.1
(Xp(PP)
63.4 +: 38.6
99.9 + 14.2
87.0 = 12.3
94.9 + 27.9
111.0 + 21.5
114.4 25.1
Exp( (xp(WORLD)
54.4 + 17.3
98.7 + 18.7
43.2 + 15.7
89.3 + 14.8
107.4 + 11.0
93.2 + 12.0
Med(CN)
19.7 8.7
49.6 = 14.9
30.8 = 7.3
45.0 + 8.8
67.9 1 16.9
70.31 10.4
Med(PP)
50.0 + 15.6
57.4 = 13.9
59.4 = 11.1
61.1 + 27.1
87.1 1 12.2
77.4 = 10.5
(WORLD)
25.7 + 21.3
33.4 + 12.8
35.6 + 6.0
54.7 + 11.0
63.6 + 8.7
55.1 + 3.5
Med-R(CN)
10.8 7.7
26.8 = 15.2
22.4 = 9.3
15.9 + 11.2
33.2 + 12.6
28.6 = 5.6
Med-R(PP)
18.3 9.5
56.3 = 16.6
44.2 = 4.5
32.5 + 15.1
69.0 + 19.3
64.3 = 7.8
(WO-RRWORLDD)
4.5 10.1
28.9 + 17.2
10.7 + 2.8
34.8 + 16.7
50.9 + 14.2
39.9 + 13.4
Rand( (CN)
12.4 1 9.1
22.9 = 10.4
6.0 + 3.1
22.2 1 4.6
32.8 = 9.5
28.13 4.6
(PP)
5.5 +
12.0 + 5.2
15.6 = 34
14.7 +
20.9 8.3
30.3 + 5.4
Rand (WORLD)
0.1 + 4.5
6.2 + 6.7
0.6 + 2.4
8.7 + 3.3
15.8 + 6.1
10.1 + 6.6
5
Experiments
Based on the above, methods include MACCA-OMAR, MACCA-CQL and MACCA-ICQ. For base- lines, we compare with both CTDE and independent learning paradigm methods, including I-CQL (Kumar
et al. 2020):
conservative Q-learning in independent paradigm, OMAR (Pan et al. 2022 based on I-CQL,
but learning better coordination actions among agents using zeroth-order optimization, MA-ICQ (Yngg
et al. 2021 Implicit constraint Q-learning within CTDE paradigm, SHAQ (Wang et al. 2022a and
SQDDPG (Wang et al., 2020): variants of crodit assignment method using Shapley value, which are the SOTA on the online multi-agent RL, SHAQ-CQL: In pursuit of more fair comparison, we integrated CQL with SHAQ, which adopts the architectural framework of SHAQ while using CQL in the estimations of agents' Q-values and the target Q-values, QMIX-CQL: conservative Q-learning within CTDE paradigm, following QMIX structure to calculate the Qhot using mixing layer, which is similar to the MA-ICQ framework. We evaluate those performance in two environments: Multi-agent Particle Environments (MPE) (Lowe et al. 2017) and StarCraft Micromanagement Challenges (SMAC) (Samvelyan et al. 2019). Through these comparative evaluations, we want to highlight the relative effectiveness and superiority of the MACCA approach. Furthermore, we conduct three ablations to investigate the interpretability and efficiency of our method. For detailed information about the environments, please refer to Appendix
5.1 General Implementation
Offline Dataset. Following the approach outlined in Justin al. (2020) and Pan al. 2002 we classify the offline datasets in all environments into four types: Random, generated by random initialization. Medium- Reply, collected from the replay buffer until the policy reaches medium performance. Medium and Expert, collected from partially trained to modcrately performing policies and fully trained policics, respectively. The difference between our setup and Pan et al. (2022) that we hide individual rewards during training and store the sum these individual rewards in the dataset as the team reward. By creating these different datasets, we aim to explore how different data qualitics affect algorithms. For MPE, we adopt the normalized score as a metric assess performance. The normalized score calculated by 100 x (S //andom))(SSsppp Srandom) following by Justin et al. (2020), where the S,Snnoom Sexpert the evaluation return from the current. policy, random set behaviour policy, expert set behaviour policy respectively.
5.2 Main Results
Multi-agent Particle Environment (MPE). We evaluate our method in three distinct environments: Cooperative Navigation (CN), Prey-and-Predator (PP), and Simple- World (WORLD). In the CN envi- ronment, three agents aim to reach targets. Observations include position, velocity, and displacements to largels and other agents. Actions are continuous in x and y. Rewards are based on distance to largets, with collision penalties. In the PP environment, three predators chase a random prey. Their state includes position, velocity, and relative displacements. Rewards are based on distance to the prey, with bonuses for captures. The WORLD environment has four allies chasing two faster adversaries. As depicted in Table I It can be seen that the algorithme hiss based on MACCA perform better than their respective backbones.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table Averaged win rale of MACCA-based algorithms and baselines in StarCraft II tasks
Map
Dataset
I-CQL
OMAR
MA-ICQ
MACCA-CQL
MACCA-OMAR
MACCA-ICQ
Expert
0.70=0009
0.86+=0.08
0.80=0.01
0,88=0.07
0.99=0.05
0.96#0.01
2s3z
Medium
0.20=0003
0.17+0.01
0.16=0.07
0.27=0.02
0.53=0.03
0.51+0.03
(Easy)
Medium- Replay
0.11=0007
0.35++0.88
0.31=0.04
0,25+0003
0.53+0.01
0.59+0.04
Expert
0.02-0.02
0.441:004
0.38100.55
0.6310002
0.7310.04
0.88.10.01
5m _s______ vs_6m (Hard)
Medium
0.01+00.00
0.14+0.02
0.11+00.44
0.19+0.001
0.20=#0.04
0.15++0.02
Medium-Replay
0.12+0.01
0.09++0.04
0.18+00.4
0.15+00.2
0.14+0.01
0.28:+0.01
Expert
0.00=0000
0.18:00.08
0.04
0.59#0.01
0.75=00007
0.60:0.03
6h_ v______ 8z
Medium
0.01:10001
0.12:10.06
0.0110.01
0.17100.00
0.20:10.02
0.22:10.04
(Super Hard)
Modiut--eppee
0.03
0.01110.01
0.07
0.1410002
0,22-10.01
0.25:10.05
MMM2 (Super Hard)
Expert
0.08=0003
0.10++0011
0.11=0.01
0.60=0.001
0.69=#0.01
0.71#0..0
Modium
0.02-1001
0.12100.2
0.0810.04
0.25+10.07
0.50100.06
0.59-10.04
Table 3: Compare with online off ooo-policy credit assignment baselines in SMAC
Map
Dataset
SHAQ
SQDDPG
SHAQ-CQL
QMIX-CQL
I-CQL
MACCA-CQL
2s3z
Expert
0.10:10.03
0.05+0.01
0.79=0.03
0.73#0.02
0.70=#0.09
0.88+:0.07
Medium
0.05=+0.03
0.07=0.01
0.24=0.01
0.22+0.03
0.20++0.3
0.27-+0.02
5m vs 6m
Expert
0.02:+0.01
0.00++0.00
0.10#0.03
0.03+0.01
0.02+#0.02
0.63+00.22
Medium
0.00++0.00
0.00++000
0.06+0.01
0.01+0.01
0.19+-0.01 0.01+0.00
6h vs_ 8%
Expert
0.00:10.00
0.00:10.00
0.02:10.1
0.00+10.00
0.00
0.59110.01
Medium
0.00++0.00
0.00++0.00
0.04+0002
0.00+0.00
0.01+0001
0.17++0.00
StarCraft Micromanagement Challenges (SMAC). In order lo show the performance in the scale scene, we specially selected maps with large number of agents. To illustrate, the map 2s3z needs to control 5 agents, including 2 Stalkers and 3 Zealots, the map ___v____ 8z needs to control 6 Hydralisks against 8 Zealots, and map MMM2 have 1 Medivac, Marauders and Marines. All experiments will run random seeds and the win rate was recorded, and the corresponding standard was calculated. Table shows the result. For most of the tasks, the MACCA-based method shows state-of-the-art performance compared to their baseline algorithms.
6h_vs______________________p__
Also, we considered testing online off- ooflpo algorithms in the offline setting. To this end, we introduced several baselines in SMAC for comparison with MACCA, as shown in Table 3 The table below shows the results of the added baselines compared to SMAC tasks. It becomes apparent that when directly applied to the offline sctting, online off-policy credit assignment algorithms consistently yicld suboptimal performance. Our empirical findings underscore that while SHAQ-CQL indeed exhibits advancements QMIX-CQL, our MACCA-CQL clinches the SOTA performance across all tasks.
5.3 Ablation Studies
The Impact of Learned Causal Structure. We varied the value of X in Eq 5 to control the density of the learned causal structure. Table 4 presents the average cumulative reward and the densily of the causal structure during the training process in the MPE-CA CN environment. The density of the causal structure 2: is calculated as Per = E मे
where i.*- represent is the value bigger than the threshold h. The results indicate that λ increases from 0 to 0.5, the causal structure becomes more sparse (par decreases), resulting in less policy improvement. This can be attributed to the fact that MACCA may not have enough states to predict individual rewards, leading to misguided policy learning accurately. Conversely, setting a relatively low X1 may result in a denser structure that incorporates redundant dimensions, hindering policy learning. Therefore, achieving a reasonable cansal structure for the reward function can improve both the convergence speed and the performance of policy training. We also provide the ablation for X2, please refer to Appendix F.4
8 E
Ek=1
s i.a->r
Table 5: Average normalized scores for ground truth individual reward comparison MPE-CN
Ground Truth Individual Reward. In the MPE CN expert dataset, we investigale the influence of ground truth individual rewards on agent policy up- dates. Two scenarios are compared: agents update policies using ground truth individual rewards (GT), and agents primarily rely on team rewards (without GT). Notably, OMAR with GT directly employs individual rewards for policy updates, while MACCA-OMAR with
OMAR MACCA-OMAR
With GT
114.9 - 2.4
113.7 2.3
Without GT
43.7 + 46.6
111.7 + 4.3
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: The mean and the standard variance of average normalized score, density rate Per of E with c:.-
diverse A at different time step t in MPE-CN.
X /1
1c4
3o4
5e4
1c5
2c5
-2.43 i (.01))
-14.871 7.71(0.90)
-12.3561 8.83(
9.8421 (8.77)
69.04 1 19. (9.69())72)
0.007
-7.8815.3600.944
(8.267227.440.477)
60.18:26.14(0.28) 3.004442.0600.34
99.78# (9.50(
111.65: 4.28(0013)
0.05
3.06.112.14(00990 -12.2013.87(0.87)
10.041 45.97(0.17)
23.61 44.18(0.11)
76.811 34. 34.48(0.10)
0.5
16..1945.33(0244)
-8.841 7.16(0.11)
16.401 21.04(0.07)
59.231 35 (6.29(
GT utilizes individual rewards as a supervisory signal, replacing team rewards in Eq 4 The results, presented in Table [ the small gap between MACCA+GT and OMAR+GT arises from the extra regularization introduced by jointly learning bhy and tr, a design choice that in fact improves generalization when ground when
truth r are unavailable. Thus, even though OMAR with GT slightly outperforms MACCA-OM GT is provided, MACCA OMAR's ability to learn ff without supervision is what enables major performance gains over baselines that rely solely on leam rewards. For further details on prediction accuracy convergence versus GT, see Appendix F.6
The Impact of Causal Graph Types. To investigate the performance under different graph types, we consider three settings. The Fully Con- nected Graph assumes all variables are causally connected, while The Fixed Graph learns static graph that is invariant to time by averaging the predicted masks ei overall time steps dur- ing training. Our proposed graph setting, as described in Equation 3 learns a graph that depends on the current state s, and action a, Table o presents the results of MACCA-OMAR under these different graph types. The Fully Connected Graph yields subopti- mal performance due to its inability to differentiate individual agent contributions. The Fixed Graph shows marginal improvement over the Fully Connected Graph but remains limited in capturing the complex dynamic multi-agent causal relationships that vary with time. In contrast, our proposed dynamic graph setting achieves the highest performance by incorporating time-varying information. Additionally, we compared the performance of our method with and without h clipping, where the threshold h filters the causal mask. The results demonstrate that our method with & clipping outperforms the variant without it. This aligns with established practices in carlier works on DAG structural learning (Zheng et al., 2018 Ng et al., (020), which show the importance of clipping to ensure edge weights converge to zero when working with finite datasets. Appendix provides additional results of MACCA under different levels of h.
Table 6: Average win rate in SMAC 5m vs 6m map, expert dataset.
Win Rate
MACCA (Fully Connected Graph)
0.38 + 0.02
MACCA (Fixed Graph)
0.50 = 0.01
MACCA (w.o h clipping)
0.66 + 0.01
MACCA (w. h clipping)
0.73 = 0.04
Visualization of Causal Structure. In Figure 3 we provide visualizations of two significant causal structures within the CN enviromment of MPE. To observe the causal structure learning process more easily, we initialize the 25 as a normalized random number close to 1 and the 25 close to O. Over time, we notice that the causal structure 2: shifts its focus from considering all dimensions of the agent state to primarily emphasizing the 4th to 10th dimensions of each agent. In this environment, the agent's 's state comprises 18 dimensions. Specifically, dimensions 0th to 4th us agent's velocity and position, 5th to 9th capture the distance between the agent and three distinct landmarks, 10th to 13th refiect the distances between the agent and other agents, and dimensions 14th to 17th are related to communication, although not applicable in this experiment and thus considered irrelevant. In other words, the dimensions 4th to 9th and 10th to 13th are intuitively linked to individual rewards, aligning with the convergence direction of MACCA. With regard to the causal structure zi-- as each agent's actions involve continuous motion without extrancous variables, it converges to relevant. states that contribute to individual credits for the team reward. The results support. the interpretability of relationships between variables through the causal structure.
Training paradigms In MACCA, we train the causal model and policy alternately rather than train the causal model at the beginning. The benefit of alternated training is that the reward model is less accurate at the early stage of training, which encour- ages agents to extract diverse behaviours that go beyond the dataset. Similar to (Hu et al., 2024)), they discuss the usefulness of random rewards prior. We conducted experiments as detailed in Table Here, the TCB stands for training the causal model at the
Table 7: The win rate and loss of different train- ing paradigms by using MACCA-OMAR in SMAC 5m vs 6m, expert dataset
Win Rate Causal Model Loss
TCB 0.62 + 0.08
0.80 + 0.02
TCPA 0.73 + 0.04
0.81 + 0.01
10

---

Published in Transactions on Machine Learning Research (06/2025)
Dimension of State
18)
Dimension of State 18)
Dimension of State 18)
14
12
10
8
6
16
14 12
10
6
4
2
16
14
12
10
8
6
4
2
Agent
1.0
34
32
30
28
26
24
22
20
18
34
32 30
28
28
24
22
20
18
34
32
30
28
26
24
22
20
18
Agent
52
50
48
46
44
42
40
38
-08
3
Timestep: 10*
Timestep: 10*
Timestep: 104
(a) Causal Structure: States Individual Rewards
Dimension of Action (3 x 2)
Dimension of Action (3 x 2) 1
Dimension of Action (3 x 2)
0
-04
Agent
Agrnt
Agent
2
Agent
Agest 2
Agent
-02
Ageat
Agest 3
Agent
Timestep: 1 10
Timestep: 3 x 10
Timestep: x 10
(b) Causal Structure: Actions to Individual Rewards
Figure 3: The figure visualizes the causal structure, showing the probability of causal edges from blue (high probability) to ycllow (low probability). (a) represents the causal structure 2, 2: between the state of all agents (18 dimensions for each agent, 54 dimensions for joint stale and the individual reward dimension for each agent). (b) represents the causal structure ạ/- between the action of each agent (2 dimensions for each agent, six dimensions for joint action) and the individual reward (1 dimension for each agent).
beginning, and the TCPA is training the causal model and policy alternately. The causal model is initially trained with the same training time steps as the alternating setting, which 10 million steps. According to the result, for both paradigms, the reward model loss converged to comparable levels, and TCPA showed a clear improvement in the win rate.
Table 8: Performance on the sparse-reward variant of MPE-CN. Despite the highly non-additive team reward, MACCA significantly improves credit. assignment and overall policy performance.
Robustness to Non-Additive Rewards. On R₂
sparsc-reward variant of MPE-CN-win only if all three agents cover distinct landmarks and R2 = 0 otherwise (so ri * 3MMCAA still outperforms the baseline (Table [), demonstrating that can learn meaningful latent individual rewards and eveective policies even when the team reward is not decomposable.
Method
Avg. Episode Reward
OMAR MACCA-OMAR
0.18 + 0.07
0.42 + 0.13
6 Conclusion
In conclusion, MACCA emerges as a valuable solution to the credit assignment problem in offline Multi-agent Reinforcement Learning (MARL), providing an interpretable and modular framework for capturing the intricate interactions within multi-agent systems. By leveraging the inherent causal structure of the system, MACCA allows us to disentangle and identify the specific credits of individual agents to team rewards. This enables us to accurately assign credit and update policies accordingly, leading to enhanced performance compared to different baseline methods. The MACCA framework empowers researchers and practitioners to gain deeper insights into the dynamics of multi-agent systems, facilitating the understanding of the causal factors that drive cooperative behavior and ultimately advancing the capabilities of MARL in a variety of real-world applications.
Limitation and Future Work. One limitation of the current work is that the experiments focused on simulated environments rather than real-world scenarios. While the MPE and SMAC environments provide coutrolled best to evaluate the approach, the performance of MACCA in practical multi-agent applications remains to be investigated. Future work could explore integrating the method with real robot systems or testing it on datasets collected from roal-world multi-agent interactions to further validate its practicality and robustness.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Acknowledge
This work was supported by the Engineering and Physical Sciences Research Council |grant number EP EP/Y003187/1]
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspoctive on offiine reinforce- ment. learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Yu-han Chang, Tracey Ho, and Leslie Kaelbling. All learning is local: Multi-agent learning in global reward games, In Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003.
Sirui Chen, Zhaowei Zhang, Yali Du, and Yaodong Yang. Stas: Spatial-temporal return decomposition for multi-agent reinforcement learning. In In Proceedings of the AAAI conference on artificial intelligence, 2023.
Christian Schroeder de Witt, Tarun Gupla, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arxi::201 .9553,, 2020.
Yali Du, Lei Han, Mong Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yali Du, Joel Z Leibo, Usman Islam, Richard Willis, and Peter Sunehag. A review of cooperation in multi-agent lcarning. arXiv preprint (xX::2312.05162, 2023.
Fan Feng, Biwci Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non-stationary rein- forcement learning. In Advances in Neural Information Processing Systems, volume 35, pp. 31957-31971. Curran Associates, Inc., 2022.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, pp. 1587-1596. PMLR, 2018
St John Grimbly, Jonathan Shock, and Arnu Pretorius. Causal multi-agent reinforcement learning: Review and open problems. In Cooperative AT Workshop, Advances in Neural Information Processing Systems, 2021.
Lewis Hammond, James Fox, Tom Everiut, Ryan Carey, Alessandro Abate, and Michael Wooldridge. Reasoning about causality in games. Artificial Intelligence, 320:103919, 2023. ISSN 0004-3702. doi: https:/ //opss///oi.rgg a00t
Hao Hu, Yiqin Yang, Jianing Ye, Ziqing Mai, and Chongjie Zhang. Unsupervised behavior extraction via random intent priors. Advances in Neural Information Processing Systems, 36, 2024.
Jiaheng Hu, Peter Stone, and Roberto Martin-Martin Causal policy gradient for whole-body mobile manipulation. In Robotics: Science and Systems XIX, 2023.
Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. In International Conference on Learning Representations, 2021.
Biwei Huang, Chaochao Lu, Liu Leqi, José Miguel Hernández-Lobato, Clark Glymour, Bernhard Schölkopf, and Kun Zhang. Action-sufficint state representation learning for control with structural constraints. In International Conference on Machine Learning, pp. 9260 02099997 PMLR, 2022.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040 3049. PMLR, 2019.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. or arXiv preprinl 0tt::1088.0892 2021.
Fu Justin, Kumar Aviral, Nachum Ofir, Tucker George, and Levine Sergey, D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint orXi:20044002110 2020.
Nya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. Tn International Conference on Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce-
ment learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179-1191. Curran Associates, Inc., 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar0iv:2005. 010.0005.0664.. 01643, 2020.
Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Dala Mining, pp. 934-942, 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compeitiyy environments. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 6382-6393, 2017.
Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, 2021.
Kevin Patrick Murphy. Dynamic bayesian networks: representation, inference and learning. University of California, Berkeley, 2002.
Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. In Advances in Neural Information Processing Systems, volume 33, pp. 17943-17954. Curran Associates, Inc., 2020.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Ling Pan; Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent. reinforcement learning with actor rectification. In International Conference on Machine Learning, pp. 17221-17237. PMLR, 2022.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2009a.
Judea Pearl. Causality. Cambridge university press, 2009b.
Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactull data
augmentation. In Advances in Neural Information Processing Systems, volume 35, pp. 18143-18155 Curran Associates, Inc., 2022.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(:)7233 7284, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ar Xiv preprint or arxiv 1902 04043, 2019.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Cousation, prediction, and search. MIT press, 2000.
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Valus-decomposition networks for cooperative multi-agent learning. In Proceedings of the 1 17th International Conference Autonomous Agents and MultiAgent Systems, 2018.
Wei-Cheng Tseng, Hhun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offiiee multi-agent rein- forcement, learning with knowledge distillation. In Advances in Newral Information Processing Systems, volume 35, pp. 226 237. Curran Associates, Inc., 2022.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.
2020.
Jianhong Wang. Yuan Zhang, Yunjie Gu, and Tae-Kyun Kim. Shaq: Incorporating shapley value theory into multi-agent q-learning. In Advances in Neural Information Processing Systems, volume 35, pp. 5941-5954, 2022a.
Mianchu Wang, Rui Yang, Xi Chen, and Meng Fang. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models. In NewrIPS 2023 Workshop on Goal-Condition Conditioned Reinforcement Learning, 2023.
Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task- independent state abstraction. In International Conference on Machine Learning, pp. 23151-2311 2022b.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2.
MIT press Cambridge, MA, 2006.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang, Rethinking goal-conditioned supervised learning and its connection to offline rl. In International Conference on Learning Representations, 2022.
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pp. 10299 10312, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 14129 14142, 2020.
Jiji Zhang and Peter Spirtes. Intervention, determinism, and the causal minimality condition. Synthese, 182 (3):335-347, 2011.
Junzhe Zhang and Elias Barcinboim. Markov docision processes with unobscrved confounders: A causal approach. Technical report, Technical report, Technical Report. R-23, Purdue AI Lab, 2016.
Junzhe Zhang, Daniel Kumor, and Elias Barcinboim. Causal imitation learning with unobserved confounders. In Advances in Neural Information Processing Systems, volume 33, pp. 12263-12274, 2020.
Yudi Zhang, Yali Du, Biwei Huang, Ziyan Wang, Jun Wang, Meng Fang, and Mykola Pechenizkiy. Interpretable reward redistribution in reinforcement learning: A causal approach. In Advances in Neural Information Processing Systems, 2023.
Yudi Zhang, Yali Du, Biwei Huang, Meng Fang, and Mykola Pechenizkiy. A causality-inspieed spatial- temporal return decomposition approach for multi-agent reinforcement learning. In NewrIPS 2024 Causal Representation Learning Workshop, 2024.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Xun Zheng, Bryon Aragam, Pradeep K. Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In Advonces in Newral Information Processing Systems, volume 31, 2018.
15

---

Published in Transactions on Machine Learning Research (06/2025)
A Broader Impact Statements
The proposed work advances offline multi-agent reinforcement learning by introducing a general approach to credit assigument that can be integrated with existing algorithms, yiekding significant performance gains in static-data settings. Beyond academic benchmarks, MACCA's identifiale causal structures offer critical value in real-world applications: for instance, multi-robot coordination in logisties or disaster-response Beams-whend understanding each robot's contribution is essential for fault diagnosis and task allocation; autonomous driving fleets where clear, interpretable altributions responsibility among vehicles improve accountability, safety, and regulatory compliance; and distributed control systems in domains such as smart grids algorithmic trading- where transparent, causally grounded decisions enhance reliability and risk management. At the same time, we recognize risks inherent in learning causal models from biased incomplete offline tettastss-isestinnee behavior in high-stakes environments. To mitigate these risks, we emphasize rigorous dataset auditing to detect sampling biases, validation of learned causal structures through simulation before deployment, and cautious incremental rollout in mission-critical applications to ensure that decisions remain safe and oquitable."
causal relations could lead to unfair credit assignment unsafe
B Reproducibility Statements
To promote transparent and accountable rescarch practices, we have prioritized the reproducibility of our method. All experiments conducted in this study adhere to controlled conditions and well-known environments and datasets, with detailed descriptions of the experimental settings available in Section 5 and Appendix F The implementation specifics for all the baseline methods and our proposed MACCA are thoroughly outlined in Section and Appendix
C Markov and Faithfulness Assumptions
A directed acyelic graph (DAG), g (V, E), can be deployed to represent graphical criterion carrying out a set of conditions on the paths, where V and E denote the set of nodes and the set of directed edges, separately.
Definition C.1. (d-separation Pearl 2009a)). A set of nodes Z S V blocks the path p if and only if (1) p contains chain m + j or a fork +- m + j such that the middle node m is Z, or (2) p contains a collider m + j such that the middle node m is not in Z and such that no descendant of m in Z. Let X, Y and Z be disjunct sets of nodes. If and only if the set Z blocks all paths from one node in X to one node in Y, Z is considered to d-separate X from Y. denoting as (X La Y Z).
Definition C.2. (Global Markov Condition (Spiress et al. 2000 Pearl 2009a ) If, for any partition X, Z), X is d-separated from Y given Z, i.e. LLeY Y 1 Z. Then the distribution P over V satisfies the global Markov condition on graph G, and can be factorizes as, P(X, Z) = P(X P Z). That is, X conditionally independent of Y given Z, writing as X 11 Y Z. Definition C.3. (Faithfulness Assumption (Spirtes et al., 2000; Pearl 2009a)). The variables, which are not entailed by the Markov Condition, are not. independent of each other.
Under the above assumptions, we can apply d-separation independencies from given DAG G. That is, for any disjoint subset of nodes and X Y Z are the necessary and sufficient condition of each other.
critcrion to understand the conditional
( Z)
D Proof of Identifiability
Proposition D.1 (Individual Reward Function Identifiability). Suppose the joint state St, joint action a; team reward R2 are observable while the individual rł for each agent are unobserved, and they are from the Dec-POMDP, described in Eg 2 Then, under the Markov condilion and failhfulness assumplion, given the current time step's 's team reward R2, all the masks cx-,,, ca-rr,, well the function f are identifiable.
Assumption We assume that, f,. in Eq are i.i.d additive noise. From the weight-space view of Gaussian Process (Williams & Rasmussen 2006 and equation 6 equivalently, the causal models for rf can be represented as follows,
rf =(c 2 8t, c
La-t 2 a) + Er.t W Wo dr ar, i) Eitt
(10)
16

---

Published in Transactions on Machine Learning Research (06/2025)
where Vi E N1, and or denote basis function sets.
ag As 8, = fs "., sy wo si i SN et and a, = {ai, d tr ... of da )). We denote the variable set in the system by V Vo. V where Vi 8; Ua, U Re: and the variables form Bayesian network G. Following AdaRL (Huang et al. 2021 there possible edges only from s/.4 E 8, to rt. and from a) E at to ri in 9, where j,j are dimension index in [l.... d and [l....dd] d respectively. In particular, the rf are unobserved, while R. = Ei= ri is observed. Thus, there are deterministic edges from each rf to R.
Proof of the Proposition B.1 We aim to prove that, given the learn reward Re, and the ch cla and ry are identifiable. Following the above assumption, we can rewrite the Eq 2 to the following,
N R₁ =
Σ
r
1=1
=
"V ar, t
(11)
w Σ or (s a., + Σ cip
For simplicity, we replace the components in Eq by.
N
Tr, Σ
(s,, at;
(12)
t, Σ ci
Consequently, we derive the following equation,
R W, (x,(())) + Ey,
(13)
where X, [ts ar, 0/ representing the concatenation of the covariates 8, a, and é, from = to N.
Then obtain closed-form solution of W, in Eq 13 by modelling the dependencies between the covariates X, and response variables Rg One classical approach to finding such solution involves minimizing the quadratic cost and incorporating weight-decay regularizer to prevent overfitting. Specifically, we define the cost function as,
1 Σ (R+ W. (x))) > Xe.RowD
C(W,)
(14)
where X, and long-term returns Re, which are sampled from the offline dataset D. À is the weight-decay regularization parameter. To find the closed-form solution, we differentiate the cost function with respect to Wy and set the derivative to zero:
OC(W,) 0.
(15)
oW,
Solving Eq 15 will yield the closod-form solution for Wf, as
Wf (Ala + 2,, 7) - 2, 2 T " + -1,)--B,
(16)
Therefore, W,, which indicates the causal structure and strength of the edge, can be identified from the observed data. In summary, given team reward RG the binary masks, ci cha-tr and individual rf are identifiable.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Considering the Markov condition and faithfulness assumption, VE, V, E V, VE and V, are not adjacent in the causal graph 9 if and only if they conditionally independent. given some subset of {Vi + k, + j}. Additionally, since there are no instantaneous causal relationships and the direction of causality can be determined an edge exists, the binary structural masks ca and cla defined over the set V are identifiable with conditional independence relationships (Huang al., Consoquently, the functions f in Equation 2 are also identifiable.
conclude that for any pair of variables
2022
E
Environments Setting
We adopt the open-source implementations for the multi-agent particle environment (Lowe et al. 2017 and Samvelln et al. 2019 2
The tasks in the multi-agent particle environments are illustrated Figures (c). The Cooperative Navigation (CN) task involves 3 agents and landmarks, requiring agents cooperate covering the landmarks without collisions. In the Predator-Prey (PP) task, predators collaborate capture prey that faster than them. Finally, the WORLD task features slower cooperating agents attempting to catch 2 faster adversaries, with the adversaries aiming to consume food while avoiding capture.
Apunt
Predetor
Prodelu S
Londrork
Londrork
Advursary 2
Agent
Tommet
Landbrakk
Adwararr
Landrrork
A
Apeita
Prodwer 3
(a) CN
(b) PP
(c) WORLD
(d) 2s3z
(e) 5m vs 6m
Figure 4: Visualization of different environment in the experiments, (a)-(c): Multi-agent Particle Environ- ments (MPE), (d)-(c): StarCraft Micromanagement Challenges (SMAC)
During training, utilize the team reward as input, while for evaluation purposes, compare the performance with the ground truth individual reward. As result, the expert and random scores for the Cooperative Navigation, Predator-Prey and World tasks are as follows: Cooperative Navigation expert: 516.526, random: 160.042; Prodator-Prey expert: 90.637, random: -25669; World expert: 34.661, random: -8.734;
Datascts.
http:///chhbb..mmmmmn//mmlaaatth eartr
httpp: https://ghhhubb.mmm/////////////
18

---

Published in Transactions on Machine Learning Research (06/2025)
F Implementations
F.1
Algorithm
A Algorithm MACCA: Multi- Causal Credit Assignment T do
for training step
Sample trajectorics from D, save in minibatch for agent = 1 to N do Update the team reward R2 to f in B (Eq 6
5:
Optimize M' 4m +- A. V 9m Lm (Eq
6:
Update policy æ with minibatch B (Eq Eq or Eq
7:
Reset 3t-
F.2 Model Structure
The parametric generative model v'm used in MACCA consists of two parts: v/ and 1/. The function of th is to predict the causal structure, which determines the relationships between the environment variables. The role of 4/r is to generate individual rewards based on the joint state and action information. This prediction is achieved through a network architecture that includes three fully-connected layers with an output size of 256, followed by an output layer with a single output. Each hidden layer is activated using the rectified lincar unit (ReLU) activation function.
During the training process, the generative model is optimized to learn the causal structure and generate individual rewards that align with the observed team rewards. The model parameters are updated using Adam, to minimize the discrepancy between the predicted sum of individual rewards and the team rewards. The training process involves iteratively adjusting the parameters to improve the accuracy of the predictions.
For a more detailed overview the training process, including the specific loss functions and optimization algorithms used, please refer Figure 2 The Figure provides step-by-step illustration of the training pipeline, helping to visualize the flow of information and the interactions between different components of the generative model.
Table 9: The Common Hyperparameters.
hyperparameters
value
hyperparameters
value
steps per update
100
optimizer learning rate
Adam
batch size
1024
10
hidden layer dim
64
y
0.95
evaluation interval
1000
evaluation episodes
10
Table 10:
Hyperparameters for OMAR, CQL and MACCA
OMAR
CQL
MACCA A1
MACCA X2
MACCA Fir
MACCA h
Expert
0.9
5.0
7e-3
7
5e-2
0.1
Medium
0.7
0.5
5e-3
5e-3
5e-2
0.1
Miedium-Replan
0.7
1.0
5c-3
7
5c-2
0.1
Random
0.99
1.0
1e-7
1e-3
5e-2
0.1
F.3 Hyper-parameters
The common hyperparameters are shown in Table The neural network used in training initialized from
scratch and optimized using the Adam optimizer with learning rate of 3 x 10-4 The policy learning process involves varying initial learning rates based on the specific algorithm, while the hyperparameters for policy learning, including discount. factor of 0195, are consistent. across all tasks.
19

---

Published in Transactions on Machine Learning Research (06/2025)
The training procedure differs across tasks. For MPE, the training duration ranges from 20,000 60,000 iterations, with longer training for behavior policies that perform poorly. The number of steps per update is set to 100.
During cach training iteration, trajectories are sampled from the offline data, and the generated individual reward replaced with the leam reward for policy updates. The training of than is performed concurrently with Vrew- Validation is conducted after each epoch, and the average metrics are computed using random seeds for reliable evaluation.
The hyperparameters specific to training MACCA models can be found in Table 10
All experiments were
conducted on high-performance computing (HPC) system featuring 128 Intel Xeon processors running 2.2 GHz, 5 TB of memory, and an Nvidia A100 PCIE-40G GPU, This computational setup ensures efficient. processing and reliable performance throughout the experiments.
F.4 Ablation for X2
We have conducted ablation experiments on X2 and show the results in the Table 11
Table 11: The mean and the standard variance of average normalized score, sparsity rate Par of cia- with
diverse A2 at different time step in MPE-CN.
A /
le4
5e4
le5
2e5
17.4
(0.9().98)
93.1 + 6.4 (1.0)
105 3.5 (1.0)
107.7 10.2 (1.0)
0.007
19.9 + 12.4 (0.8
90.2 7.1 (1.0)
108.8 + 4.0 (1.0)
111.7 4.3(1.0)
0.5
13.3 = 11.1 (0.68)
100.5 + 14.0 (0.84)
102.9 + 16.4 (0.87)
108.4 + 6.4 (0.98)
5.0
2.3 + 9.8 (0.0)
-1.3 + 25.4 (0.34)
70.4 + 18.0 (0.62)
100.1 + 7.4 (0.75)
F.5
Ablation for
The selection of l can influence the sparsity of the causal graph. h can be selected by parameter sweeping. For simplicity, we use h = 0.1 for all tasks in the experiments, which leads to strong performance. we conduct additional experiments under different h in SMAC 5m_vs vs 6m Medium Dataset with MACCA-OMAR. The results are as follows,
Table 12: The mean and the standard variance of the average normalized score, sparsity rate Par of aja-v
with diverse h in SMAC 5m vs 6m.
h
Win Rate
Psr
Par
Causal Model Loss
0
0.12 + 0.02
1.0 + 0.0
1.0 = 0.0
0.15 + 0.05
0.01
0.14
+0.03
0.96 = 0.12
0.72 = 0.12
0.07 + 0.01
0.05
0.16 +0.02
0.81 + 0.07
0.66 + 0.04
0.09 + 0.04
0.1
0.20 + 0.04
0.73 + 0.04
0.54+ 0.08
0.05 + 0.02
0.5
0.17 + 0.01
0.52 = 0.10
0.43 = 0.07
0.12 + 0.06
The causal graph become more sparse (fewer edges between nodes) with the increase of The performance of win rate goes up with the increase of h but decrease after 0.1, due to potential inclusion of redudance information.
F.6 Prediction Accuracy After Convergence (By Dataset Quality)
After convergence, we measure per-agent errors between redistributed rewards P and true rf. Table shows MSE and MAE (averaged over time and three random seeds) for Expert, Medium, Medium- RRelay, and Random datasets.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Table 13: Per-Agent Prediction Errors After Convergence
Dataset Quality
MSE
MAE
Expert
0.12 = 0.02
0.65 + 0.10
Modium
0.24 = 0.05
1.12 + 0.15
Medium -leply
0.38 = 0.08
1.78 = 0.20
Random
1.60 + 0.12
4.45 + 0.30
These results indicate that, once MACCA has converged, the redistributed rewards closely track the true individual rewards in high-quality data (Expert), with an MSE of only 0.12 and MAE of 0.65. As dataset quality degrades Mediumm, Medium-Replay, and Random the errors increase proportionally, reflecting noisier or more random behavior, yet remain within reasonable bounds. Even on the Random dataset, where behavior is least structured, MAE of 2.45 demonstrates that 1ù still captures meaningful individual-reward signals after training.
F.7 Computational Resources and Training Times
All experiments were conducted on a hetcrogencous computing cluster running Ubuntu Linux. The hardware configuration included mix of CPU models (Dual Intel Xeon E5-2650, E5-2680 v2, and E5-2690 v3) with total of 180 CPU cores and 500 GB of system memory. For GPU acceleration, we utilized three NVIDIA A30 GPUs.
The average wall-clock training time (for 10 million environment steps) of each MACCA variant is summarized in Table 14 These timings include both causal model learning and policy optimization under the alternating training scheme; the causal model component accounts for approximately 15% of the total time.
Environment
MACCA Variant
Training Time (hrs)
MPE (CN, PP, WORLD)
MACCA-OMAR MACCA-OMAR
4.2
SMAC (2s3z)
7.8
SMAC (5m 6m)
MACCA-OMAR
9.5
SMAC (6h vs 8z)
MACCA-OMAR
11.2
SMAC (MMM2)
MACCA-OMAR
12.0
Table 14: Average wall-clock training time for each MACCA-OMAR run (10 million environment steps).
21


Reviews:
[{"review_id": "2VMBxzYP10", "reviewer": "Reviewer_rSnL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n- The idea of complementing offline MARL algorithms with learning causal structures on the joint reward function to assign individual rewards is interesting and (to the best of my knowledge) novel.\n- The experiments are thorough and clear, and help highlight the applicability and benefits of the method.\n- The method seems flexible and applicable across offline MARL algorithms.\n\n# Weaknesses\n- The paper overall lacks a strong motivation for the method proposed. In particular, it is not properly highlighted why the solution requires a causal model. One could argue that MDPs are causal by nature, and it is not clear why a simple generative model to predict individual rewards is not enough.\n- Some of the formal definitions and statements lack proper introduction, some concepts are not defined and some of the technical statements are not clear. See below for requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to improve the performance of offline MARL algorithms through credit assignment in settings where only common rewards are available. The authors introduce a framework that employs causal modeling via a Dynamic Bayesian Network (DBN) to decompose team rewards into individual rewards. They model the underlying generative process of rewards and establish theoretical identifiability results under standard assumptions (e.g., Markov condition and faithfulness), and aim to accurately assign credit to individual agents. They propose how to incorporate their method into existing offline MARL algorithms and provide extensive experiments on simulated environments (MPE and SMAC). The experiments show that MACCA-based methods outperform several state-of-the-art baselines, and ablation studies support the benefits of learning a dynamic causal structure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is not clear why is there a need for a causal model to solve the problem. Authors should motivate this further.\n2. Before eq (1), authors refer to the causal model as $\\psi_m$. Do they mean $\\phi_r$? What is $\\psi_{\\pi}$? Further along the paper they also introduce $\\psi_g$. It is not clear how all these causal models are related. A paragraph stating clearly the specific details of all these would be necessary.\n4. In eq (1) the loss $L_m$ is mentioned. I understand the authors define it later, but a brief introduction of how it looks like would be good.\n5. The authors seem to assume implicitly that one can separate the influence of each state and action component in the rewards. Is this an assumption? Can one assume this in general?\n6. In pg 5, $\\epsilon$ is referred to as 'the i.i.d. noise'. What is this noise? I believe it has not been introduced.\n7. Why are the masks needed? Is it simply to reduce model complexity?\n8. In pg 4, it is not clear how the DBN graph is constructed. Authors should expand on this, and on how it is used in practice.\n9. Before eq (3), the authors state that the masks are described by eq (2), but I believe this equation does not really describe the masks.\n10. Further discussion and relation to recent work on causality in games would be beneficial. See [1] below.\n\n\n[1] Hammond, Lewis, et al. \"Reasoning about causality in games.\" Artificial Intelligence 320 (2023): 103919.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z2SCLBrXmV", "reviewer": "Reviewer_WTmc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a novel technique of estimating the individual rewards based on the probabilistic graphical model framework. It also provides an extensive analysis of how each component of MACCA affects the final performance. However, the underlying assumption of MACCA, which is that the team reward is the sum of all individual rewards, is taken for granted without justification. This seems to be incorrect in general. For example, consider the sparse-reward version of MPE-CN, where the team reward is one if all three agents have covered the landmarks and zero otherwise. In such a case, the team reward is not the sum but the product of three individual rewards.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new framework for offline multi-agent RL called Multi-Agent Causal Credit Assignment (MACCA). MACCA utilizes a dynamic Bayesian network to infer each agent's individual reward based on the probabilistic graphical model of the environment. The authors also provide a theoretical guarantee that the graphical structure can be identified. Finally, the paper presents empirical evidence that MACCA can boost the performance of existing MARL algorithms on both discrete and continuous action settings, thereby showing its effectiveness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Justify the assumption that the team reward can be represented as the sum of individual rewards. Empirical justification, for example, showing that MACCA improves the performance of baseline algorithms in environments where the team reward and the individual rewards are related in a highly nonlinear manner, may suffice. (Critical)\n2. Provide an analysis of how close the individual rewards predicted by MACCA are to the ground truths. (Non-critical)\n3. The **Ground Truth Individual Reward** section on page 8 is difficult to follow and seems extraneous. We already know that OMAR performs poorly without GT individual rewards, and MACCA can significantly boost its performance. The fact that OMAR with GT performs well, in fact, even better than MACCA-OMAR with GT, only weakens the paper's overall claim. (Non-critical)\n4. Tables 4 and 10 are challenging to digest. Visualizing the data with graphs would make it more accessible. (Non-critical)", "broader_impact_concerns": "I do not have any concern on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tZhvh8jeQo", "reviewer": "Reviewer_6i3R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe causal modeling framework provides a fresh perspective on credit assignment in offline MARL, addressing a known challenge in the field. The identifiability proofs establish a solid theoretical basis, which is rare in applied RL research. The experimental section includes multiple environments with varying difficulty levels, showing consistent performance improvements. The framework's ability to integrate with existing MARL methods increases its practical utility. The detailed ablation analyses help clarify the importance of different components in the proposed method.\n\n**Weaknesses:**\n\nThe causal modeling approach might not be uniquely applicable to offline MARL and could be relevant to online RL settings as well. The paper could better justify why this approach is particularly suited for offline environments. No concrete code or pseudocode is provided, making it difficult to fully reproduce or extend the work. While a Broader Impact Statement is included, it could be more comprehensive regarding ethical considerations and potential misapplications. The paper doesn't adequately address the computational complexity and resource requirements of the proposed method. Most experiments are conducted in simulated environments, with limited discussion of how the method would perform in real-world scenarios. The dynamic causal structure learning may introduce computational costs, but the paper lacks an analysis of scalability or runtime efficiency.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents MACCA, a novel framework for offline multi-agent reinforcement learning (MARL) that addresses the credit assignment problem through causal modeling. The key contributions are the introduction of a Dynamic Bayesian Network (DBN) to decompose team rewards into individual agent rewards, theoretical proofs establishing identifiability of causal structures and individual reward functions in offline settings, and empirical validation showing superior performance over state-of-the-art methods across multiple environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Revisions:**\n- The authors should provide a clearer justification for why the causal modeling approach is particularly suited for offline MARL rather than being a general RL approach. This could include a discussion of how offline data constraints influence the causal structure learning process. \n- Provide code and detailed hyperparameters to ensure reproducibility.\n\n**Non-Critical Improvements:**\n- The paper should include an analysis of computational requirements, memory usage, and training time. \n- Adding comparative analysis with online MARL methods would help justify the focus on offline settings.\n- Report training time/runtime metrics for large-scale environments.", "broader_impact_concerns": "The current Broader Impact Statement mentions improving algorithm credibility and explainability but could be strengthened by discussing potential applications in real-world multi-agent systems that might affect decision-making transparency in complex systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the manuscript considered an interesting problem of credit assignment under collaborative multi-agent reinforcement learning setting with a sound solution through the causal identification framework. There are some concerns on the assumptions of the causal identifiability, but the reviewers believe that rebuttals have properly addressed the concerns. The reviewers are positive towards acceptance, but urge the authors should make a significant revision on the contents as mentioned in the review and rebuttal.
Decision: Accept (poster)