Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making
Authors: Stelios Triantafyllou
Aleksa Sukovic
Yasaman Zolfimoselo
Goran Radanovic
Venue: ICML 2025 poster (2025)
Keywords: counterfactual reasoning
causal explanation formula
multi-agent Markov decision processes
accountability
Primary area: general_machine_learning->causality
TL;DR: We address the challenge of explaining the total counterfactual effect of an agent’s action on the outcome of a realized scenario in multi-agent Markov decision processes.

Abstract: We address the challenge of explaining counterfactual outcomes in multi-agent Markov decision processes. In particular, we aim to explain the total counterfactual effect of an agent's action on the outcome of a realized scenario through its influence on the environment dynamics and the agents' behavior. To achieve this, we introduce a novel causal explanation formula that decomposes the counterfactual effect by attributing to each agent and state variable a score reflecting their respective contributions to the effect. First, we show that the total counterfactual effect of an agent's action can be decomposed into two components: one measuring the effect that propagates through all subsequent agents' actions and another related to the effect that propagates through the state transitions. Building on recent advancements in causal contribution analysis, we further decompose these two effects as follows. For the former, we consider agent-specific effects -- a causal concept that quantifies the counterfactual effect of an agent's action that propagates through a subset of agents. Based on this notion, we use Shapley value to attribute the effect to individual agents. For the latter, we consider the concept of structure-preserving interventions and attribute the effect to state variables based on their "intrinsic'' contributions. Through extensive experimentation, we demonstrate the interpretability of our approach in a Gridworld environment with LLM-assisted agents and a sepsis management simulator.

Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making
Stelios Triantafyllou Aleksa Sukovic'
Yasaman Zolfimoselo Goran Radanovic
Abstract
scenarios is fundamental for accountability, For instance, counterfactual reasoning can be employed to identify ac- tual causes (Halpern, 2016; Triantafyllou et al., 2022), at- tribute responsibiliity (Chockler & Halpern, 2004; Frieden- berg & Halpern, 2019), generate explanations (Madumal et al., 2020), evaluate fairness (Kusner et al., 2017: Huang et al., 2022) and measure harm (Richens al., 2022; Beck- ers et al., 2022). To achieve such objectives, many studies often rely on the notion of total counterfactual effects, which quantifies the extent to which an alternative action would have affected the outcome of a realized scenario.
We address the challenge explaining counter- factual outcomes multi-agent Markov decision processes. In particular, we aim explain the to- tal counterfactual effect of an agent's 's action the outcome of realized scenario through its influ- ence on the environment dynamics and the agents' behavior. To achieve this, we introduce a novel causal explanation formula that decomposes the counterfactual effect by attributing to each agent and state variable score reflecting their respec- tive contributions to the effect. First, we show that the total counterfactual effect of an agent's action can be decomposed into two components: one measuring the effect that propagates through all subsequent agents' actions and another related to the effect that propagates through the state transitions. Building on recent advancements in causal contribution analysis, further decom- pose these two effects as follows. For the former, we consider specifcc effects a causal con- cept that quantifies the counterfactual effect of an agent's action that propagates through a subset of agents. Based on this notion, we use Shapley value to attribute the effect individual agents. For the latter, we consider the concept of structure- preserving interventions and attribute the effect to state variables based on their "intrinsic" contri- butions. Through extensive experimentation, we demonstrate the interpretability of our approach in a Gridworld environment with LLM-assisted agents and a sepsis management simulator.
In multi-agent sequential decision making, an agent's ac- tion typically affects the outcome indirectly. To illustrate this, consider the problem of Al-assisted decision making in healthcare (Lynn, 2019), where a clinician and their Al assistant treat patient over period of time. Fig. 1a de- picts a specific example, where treatment fails, We estimate that the clinician had not followed the Al's recommenda- tion at step 10 and administered vasopressors (V) instead of mechanical ventilation (E), the treatment would have been successful with 82% likelihood. Therefore, the consid- ered alternative action admits a high total counterfactual effect. This effect, however, propagates through all subse- quent actions of the clinician and the AI, as well as all the changes in the patient's 's state. This makes the interpretabil- ity of the effect more nuanced, as the change from action to outcome can be transmitted by multiple distinct causal mechanisms. In this work, we ask:
How explain an action's total counterfactual effect in multi-agen sequential decision making?
Much prior work in causality has focused on decomposing causal effects (Pearl, 2001; Zhang & Bareinboim, 2018a:b) under the rubric of mediation analysis (Imai et al., 2010; 2011: Hicks & Tingley, 2011; VanderWeele, 2016), which aims understand how effects propagate through causal paths. However, such approach would not yield inter- pretability in multi-agent sequential decision making. There can be exponentially many paths connecting an action to the outcome, and not all of them have a clear operational meaning to help explain the effect intuitively. We instead posit that more natural interpret the effect of an ac- tion in terms of its influence the agents' behavior and the
1. Introduction
Applying counterfactual reasoning to retrospectively an- alyze the impact of different actions in decision making
Max Planck Institute for Software Systems, Germany, Corre- spondence to: Stelios Triantafyllou <strianta@ mpi-mms.oo ow..g....
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
State 14
Patient Vitals
Patient Vitals
Patient Vitals
Glucose: High
Glucose: HighTV. High
Glucose: Normal
Heart rate: Normal
Heart rate: Normal
Heart rate: High|Norma
Blood Pt:: Normal
Blood Ph: Normal
Blood Ph: High| Normal
Oxygen: Low
Oxygen:
Oxygen: Normal
Actions
Actions
Actions
Al:
ABE
A&E A&ERV
Al:
Clinician:
Accept|A88
Cliiciia:
Clinician:
State 13
Outcome:
Outcome:
Outcome:
Fallure Succoss
Clivican
Step
Step 11
(a) Factual and Counterfactual Trajectory
(b) Decomposition
Figure 1: Fig. la depicts (part of) a simulated scenario from the two-agent Sepsis environment in Section 6.2, where the patient's 's treatment fails. In the same figure, we have also included the values from sampled counterfactual scenario (values that are different are shown in orange), where the clinician's 's action is fixed to override the Al's action at step 10. Hence, the patient receives treatment A&V V instead of A&E. Plot lb shows the results of our decomposition approach for this scenario.
environment dynamics. Therefore, we need to analyze how the effect propagates through: (a) the subsequent agents' actions and (b) the state transitions of the environment. In the previous example, the total counterfactual effect of the considered action can be decomposed as shown in Plot lb. This approach explains the effect by attributing a score to each doctor (clinician and AI) and patient state, reflecting their respective contributions to the overall effect.
regarding the counterfactual outcomes needed for the com- putation of r-SSE. We propose method for decomposing r-SSE that is efficcent under relatively mild assumption that at least one state variable has non-zero ICC (i.e., is informative about the counterfactual outcomes).
We experimentally validate the interpretability of our ap- proach using two multi-agent environments: a grid-world environment, where two RL actors are instructed by LLM planner complete sequence of tasks, and the sepsis man- agement simulator from Fig. 1.¹
Contributions. Focusing on Multi-Agent Markov Decision Processes and Structural Causal Models, provide sys- tematic approach to attributing the total counterfactual effect of an agent's ss action on the outcome of given trajectory, based on the following bi-level decomposition.
1.1. Additional Related Work
This paper is related works mediation analysis and especially to those that consider multiple (sequential) medi- ators (Daniel et al., 2015; Steen et al., 2017; VanderWeele & Vansteelandt, 2014; Chiappa, 2019). As mentioned car- lier, the main distinction between this line of work and ours is that we analyze how effects propagate through agents and state variables in an MMDP, instead of causal paths in general SCMs. In similar sense, our work also relates the areas of causal contributions (Janzing et al., 2024; Jung et al., 2022; Heskes et al., 2020) and flow-based attribution methods (Singal et al., 20 Wang et al., 2021). The former studies how to attribute target effect to different causes (of- ten model features) based on their degree some notion of contribution that effect. The latter considers the problem of assigning credit the edges of causal graph, instead of the nodes, for explaining causal effects.
(Level 1) We first introduce causal explanation formula, which decomposes the total counterfactual effect of agent's action into the total agent-specific effect and the reverse state-specific effect. The former refers to the effect that propagates through all subsequent agents' actions, and is formulated via the recently introduced notion of agent- specific effects (Triantafyllou et al., 2024). The latter refers to the effect that would have been lost or gained had the action not been propagated through the state transitions, and it is special case of path-specific effects (Avin et al., 2005).
(Level 2a) To further decompose the total agent-specific effect (tot-ASE), we propose an axiomatic framework based on agent-specific effects for attributing the total effect to individual agents. The set of axioms includes efficiency, which requires that agents' contributions sum up to tot-ASE. We show how to operationalize Shapley value with agent- specific effects, obtain method for decomposing tot- ASE, which uniquely satisfies the set of proposed axioms.
2. Background and Formal F Framework
In this section, we present our formal framework, which is adopted from (Triantafyllou et al., 2024) and builds on Multi-
(Level 2b) To further decompose the reverse state-specific effect (r-SSE), we utilize the notion of intrinsic causal con- tributions (ICC) (Janzing et al., 2024). ICC enables us to quantify the informativeness of individual state variables
'ode to reproduce our experiments is available at http:///gghhbbccoomsccllittttfff decp:fffcccddooooooiiiiio..ooo
2

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Agent Markov Decision Processes (MMDPs) (Boutiller,, 1996) and Structural Causal Models (SCMs) (Pearl, 2009). A table summarizing the notation is provided in Appendix B. Appendix N provides graphical illustration of all counter- factual effects discussed in this section and the next.
similar to general SCMs, the MMDP-SCM induces di- rected causal graph, which can be found in Appendix D. Appendix C also describes the conditions under which the observational distribution of an MMDP- MMDDP-CCM is consistent with some MMDP and joint policy. In this paper, we focus on categorical MMDP-SCMs.
2.1. Multi-Agent Markov Decision Processes
2.3. Interventions and Counterfactuals
An MMDP is represented as a tuple (S, (1, n), A,T,
o), where: S is the state space; {1,..., n. is the set of agents; A = x==1" is the joint action space, with A, being the action space of agent i; T S x A x S - [o, 1] is the tran- sition probability function; is the finite time horizon; or is the initial state distribution.2 Fach agent i E (1,...n) n has stationary decision-making policy ni. with the joint policy of all agents represented as πT The probability of agents jointly taking action a, = (a1.s...., Rn,s) in state 8, at time t is thus given by mayys))) = 11 (A1_0(() ... Tn (an.t (st), while the probability of transitioning from state 8t to state s141 is determined by [(s at). A sequence of such state-action pairs {(s,, (E)).. and final state 8h is called trajectory. With 5 r(X), we denote the value of variaale X in trajectory r.
Consider an MMDP-SCM M. An intervention on the action variable As, of M corresponds to the process of modifying the structural equation A == f (S', , U....) from Eq. 1. More specifically, a hard intervention do(A,, := a.,z) fixes the value of Ai, to the constant 64,t, resulting in new MMDP-SCM denoted by ddo(A,, Similar to (Correa et al., 2021), when random variables have subscripts we will use square brackets to denote interventions.
Let now Z E V and u ~ P(u). We denote with Zdo(At,:=a,.)) (u) or Zar, (u) for short, the solution of Z for u in Mdd(A,.==a.)) and with Za., the random variable induced by averaging over U. Typically, Za, (u) is re- ferred to as the potential response of Z to do(A == a,,)). A natural intervention do(A,,, == (j,,(( replaces the structural equation of An,'' in M with the potential response of Ag. to the (hard) intervention do(A,, := az,))
2.2. MMDPs and Structural Causal Models
We utilize the MMDP-SCM framework (Triantafyllou et al., 2024) to express an MMDP coupled with joint policy π as an SCM. Specifically, an MMDP- (V, U, F(
Given a trajectory T and a response variable Y E V. the counterfactual probability P(Y... = y/T) or PT ar for short, measures the probability of Y taking the value y in T had A,, been set to a,2, Subscript M here implies that the probability defined over the MMDP-SCM. M. When necessary, P(-IT; M) is used denote that was generated by M. Next, define standard causal notion that is used to quantify the counterfactual impact of intervention do(A,, == az,r) on Y.3
consists of:
(i) a tuple V = (Sp, A1,,,..,,AAA, .... Sn)) of the ob- served variables whose causal relations modeled, i.c., all state and action variables of the MMDP;
(ii) a tuple U SU (U,S, ...., UAn..,... ..... U of mu- tually independent unobserved noise variables which capture any underlying stochasticity of the MMDP and agents' policies;
Definition 2.1 (TCFE). Given an MMDP-SCM M and a trajectory T of M, the total counterfactual effect of inter- vention do(Ar, = os,t) on Y E V, relative to reference T(A,.), is defined as
(iii) A joint probability distribution P(u) = Iw P(u") over U:
TCFE R. (A. (Y17MM = E|Y :alam=E]aa,,, (1)0--B))) =E(a,, 7(Y).
(iv) A collection F of deterministic functions that deter- mine the values of all observed variables in V via the following structural equations
Assumptions and counterfactual identifiability. Note that there might be multiple MMDP-SCMs whose observational distribution is consistent with some MMDP-joint policy pair, but yield different counterfactuals, e.g., different val- ues for TCFE. This means that without further assumptions, counterfactuals cannot be identified from observations alone. To enable counterfactual identifiability. we thus make the
So = (U))) S f== AL,, A,, == (St (S,)))
(I)
Note that any context u ~ P(u) induces a unique trajectory T, such that VX E V it holds that 7 =(X is the solution of
X, for the particular u, in the MMDP-SCM. Furthermore,
IIn this paper, we consider counterfactual effects mostly rela- tive to the factual action 7 r(A,,). However, we note that generally any valid action can be used as a reference value.
Forr ease of notation, rewards are considered part of the states.
3

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
following assumptions. First, we consider unobserved vari- ables to be mutually independent. Second, we assume that MMDP-SCMs satisfy the (weak) noise monotonicity condi- tion introduced by (Triantafyllou et al.. 2024). Note that the latter assumption is not limiting for the MMDP distribution or the agents' policies, i.c., every MMDP can be consistently represented by noise-monotonic MMDP-SCM. What is restricted instead is the expressivity of the model's counter- factual distribution. Details about noise monotonicity can be found in Appendix E.
are fixed to their factual values. i.e., their actions in T. Defi- nition 3.1 is in line with the notion of agent-specific effects introduced by (Triantafyllou et al., 2024) and revisited here in Section 5, while SSE can be seen as a special case of path-specific effects (Avin et al., 2005).
Perhaps counter to intuition, it does not hold that TCFE is always decomposed into tot-ASE and SSE. An empirical counter-example for this is provided in Section 6.1. How- ever, by comparing Definitions 3.1 and 3.2, observe that the total agent-specific effect associated with the transition from the factual action T r(A,,) to the counterfactual action ai is closely related to the state-specific effect associated with the reverse transition, i.e., the effect
3. Decomposing Total Counterfactual Effect
The total counterfactual effect can inform us about the ex- tent to which an alternative action would have affected the outcome of a trajectory. However, this measure alone does not provide any further insights on why or how that action would have affected the outcome. In this section, we intro- duce novel causal explanation formula that decomposes TCFE w.r.t. the two building blocks of an MMDP - its states and its agents. First, based on prior work we define two causal quantities for measuring how much of the total counterfactual effect of agent's action on some response variable is mediated by (a) all future agents' actions and (b) the subsequent MMDP's s state transitions.
SSF (AF....) (YIT)M
(AAY(( Ma - FE(
= =T] M Maucr: - (VVa,,,TJM (2)
where I= {Ar.e == Ay.e(an (F.(....))))
We will
refer to the latter as the reverse stale-specific effect or r-SSE for short, to clearly distinguish it from SSE. In words, r- SSE measures the difference in the counterfactual value of Y under intervention do(A,, == ag,e), assuming that the state S4+1 had not been affected by the intervention, but all subsequent agents' actions had. Based on this obser- vation, we derive the following decomposition of the total counterfactual effect.
Definition 3.1 (tot-ASE). Given an MMDP-SCM M and trajectory T of M. the total agent-specific effect of inter- vention do(Ar, == a.,z) on Y E V, relative to reference
Theorem 3.3. The total counterfactual effect, total agent- specific effect and reverse state-specific effect obey the fol- lowing relationship
T rAA,,), is defined as
ASE (1...n) (Y(T)M = (As,)
(CFFa.. (YI))
LE [T; (A
= -ASE (AL,)(A
(
(Y SSE (Y17). (3)
E| [T; Mlaaaaon - -XY),
Theorem 3.3 states that the total counterfactual effect of do(A,, := as,e) on Y equals to the effect that propagates only through the agents minus the effect that would have been lost or gained had the intervention not been propagated through the states.
where / = (Av,r- =- Av. Arr.
Definition 3.2 (SSE). Given an MMDP-SCM M and a trajectory 7 of M, the state-specific effect of intervention do(A,, := a; on Y relative to reference T r(Ai,), is
defined as
Connection to prior work. Our result is similar in principle with the well-known causal mediation formula for arbitrary SCMs (Pearl, 2001), which decomposes the total causal effect of an intervention into the natural direct and indirect effects. Thus, Theorem 3.3 can be viewed as an extension of Theorem 3 from (Pearl, 2001), applied to the problem of counterfactual effect decomposition in multi-agent MDPs.
SSE (SEa,,iT(AL.)) (Y =
= =EYaa,, [T; M] M - E E(
=EYon. (y; MD Maddon - -(Y),
where 7 -= (Av == A
In words, Definition 3.1 measures the difference between the factual value of Y, i.e., 7 rYY), and the (expected) counter- factual value of Y had all agents taken the actions that they would naturally take under intervention do(A,, == ag,z) af- ter time-step t. On the other hand, Definition 3.2 measures the counterfactual effect of intervention (o,A,, := ar,t) on Y in a modified model where all subsequent agents' actions
Sepsis example. Going back to our example scenario from the introduction, the result of our decomposition can be interpreted as follows: (a) 45.6% of the TCFE is attributed to how the AI and the clinician would have responded to the intervention (tot-ASE R 0.374); (b) the remaining 54.4% is attributed to the influence that the intervention has on the patient state (-r-SSE 0.446).
4

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
4. Decomposing Reverse State-Specific Effect
case where we can exactly compute (SEr(A.))... (Y M without conditioning on any noise term, e.g., if environment and policies are deterministic, then our approach does not decompose the r-SSE any further.
In this section, we focus on further decomposing the re- verse state-specific effect. More specifically, our goal is to attribute to each state variable a score reflecting its contribu- tion to r-SSE. Our approach utilizes the notion of intrinsic causal contributions (ICC) introduced by (Janzing et al., 2024). For general SCMs, the ICC of an observed variable X to target variable Y measures the reduction of uncer- tainty in Y when conditioning on the noise variable U* In our work, we model uncertainty using the expected condi- tional variance and modify the ICC definition to quantify the influence of state variables on the variation of r-SSE.
A Agorithm. Appendix G includes an algorithm for the approximation of the expected conditional variance of aYr Our algorithm follows the standard abduction- uction--reddiction methodology for counterfactual inference (Pearl, 2009): it samples conditioning and non-conditioning noise variables independently from the posterior distribu- tion, estimates the noise- conditionl variance for r-SSE and returns the average value.
Let k E (O.....A. We denote with USK the set SU (U ,..,,AAA,, for k- < h, and with U the set (US)). We also denote with UCS the set of noise terms associated with the observed variables preceding (chrono- logically) Sk- Note that r-SSE essentially measures the expected value of the difference AYi,ar, Yr Yo.., in M. when noise terms U are sampled from the posterior distribution P(u(r). Thus, the ICC can be defined in our context as follows
Causal interpretation. The attribution method described in Definition 4.1 relies on do interventions that are performed only on agents' actions, meaning that the causal mechanisms of the environment remain intact. Conditioning on noise terms can be considered as form of ptreecturepreservvnng interventions, i.e., interventions that depend on the values of the parents of the exposure variable (here previous state and actions), and do not perturb the observed distribution. For a more detailed discussion on the causal meaning of ICC we refer the reader Section 3.1 in (Janzing et al., 2024).
ICC(S& + lT,,,, = Unc - Un,
(4)
Plain ICC. Since there is unique causal order among the state variables of an MMDP-SCU the time order, there is no arbitrariness due to order-dependence. Thus, we consider the "plain" ICC for our approach instead of the Shapley based symmeterization used in (Janzing et al., 2024).
AY1,aa,, where Une ([ IT,
Une = E[Var( AY a., |T, U U [r],
and I = (Ar.e == (y.er (reee..reeeee............. (e(......)))) "), In words, Eq. 4 measures the reduction of variance in AYr,a,, caused by conditioning on the noise variables associated with state Sy and the agents' actions taken therein. Based on this, we can now define our attribution method for r-SSE.
Finally, we show that r-SSE-ICC fully allocates r-SSE among the states following the intervention.
Theorem 4.2. Let ty denote the time-step of response variable Y and va be the output of r-SSE-ICC for the reverse state-specific effect SSF. (Y|T)M- If Var Bar(AYi,aa, IT) > 0, then it holds that Exe Us. SSE,(A..)aa., (Y)
Definition 4.1 (r-SSE-ICC). Given an MMDP-SCM M and a trajectory J of M, r-SSE-ICC assigns to each state variable S₃ for & E {0,.... h) a contribution score for the reverse state-specific effect (SE, (Y)T), equal to
Us. == ICC(S, + AYT,,,.T)MM SSE (Y)T)a,
Sepsis example. The result of the r-SSE-ICC method in the Sepsis example can be interpreted as follows: if we knew the counterfactual state of the patient at step 13, following the intervention step 10, we could estimate the reverse state- specific effect with almost no uncertainty, While, knowing the exact counterfactual value for any of the previous states would not lead to a comparable reduction in uncertainty.
Var(AYr,a, aa,.. (T)
if the unconditional variance Var(AY),.,,.17)) > 0, and equal to 0 otherwise.
According to Definition 4.1, the reverse state-specific effect is allocated among state variables in proportion to their in- trinsic contribution to the effect. Intuitively, this means that the influence of a state variable to the r-SSE is represented by the relative degree to which we can more precisely estimate the effect if we could also predict the counterfactual value of that state under the interventions do( /) and (A,,, := oi,)). Thus, if knowing what would have happened in state Sş is pivotal for the accuracy of our counterfactual prediction then contribution score d.s would be high, whereas if it has small influence then sss. would be closer to zero. In the
5. Decomposing Total Agent-Specific Effect
In this section, we focus on further decomposing the total agent-specific effect. More specifically, our goal is to at- tribute to each agent a score reflecting its contribution to tot-ASE. Our approach is based on a well-established solu- tion concept in cooperative game theory, the Shapley value (Shapley, 1953), and it utilizes the notion of agent-specifie effects introduced by (Triantafyllou et al., 2024).
5

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Definition 5.1 (ASE), Given an MMDP-SCM M, a non- empty subset of agents N in M and trajectory of M, the N-specific effect of intervention do(Ai, ;= di.e) Y E V, relative to reference T rAA,)), is defined as
Sepsis example. ASE-SV attributes tot-ASE to both the Al and the clinician. As illustrated in Plot 1b, larger portion of the effect is attributed to how the AI would have responded to the intervention.
ASE
oza (As,) (Y(T)M- =
6. Experiments
= [T] MMl (A
In this section, we empirically evaluate our approach to counterfactual effect decomposition using two environ- ments, Gridworld and Sepsis. We refer the reader to Ap- pendix J for more details on our experimental setup and implementation, and to Appendix K for additional results. Throughout both experiments, we use 100 posterior sam- ples for estimating counterfactual effects and 20 additional ones for the conditional variance. Additional experiments evaluating the estimation error of our results as well as their robustness to the noise monotonicity assumption are provided in Appendix I. and Appendix M, respectively.
= E| 17; Mlaarmo -
where 1 = {Are == Ag Ap,.(as..)) (PN,
U {Av, 1=
In contrast to the total ASE, the N-specific effect quantifies the counterfactual effect of an intervention that propagates only through subset of agents in the system, the effect agents, instead of all agents. Compared to Definition 3.1, here the actions of the non-effect agents are set to their fac- tual values. In the context of agent-specific effects studied here, Shapley value can be defined as follows.
Definition 5.2 (ASE-SV). Given an MMDP-SCM M and a trajectory a of M, ASE-SV assigns to each agent j E {I, n contribution score for the total agent-specific effect ASE().....)) (A;,)((As,)) (Y|T),, equal to
6.1. Gridworld with LLM-assisted RL agents
Environment. We consider the gridworld depicted in Fig. 2a, where two actors, Ai and A2, are tasked with delivering objects. In the beginning of each trajectory, two randomly sampled objects spawn in each of the boxes lo- cated on the rightmost corners of the gridworld. The color of each object determines its value. Colored cells indicate areas of large stochastic penalty, which is significantly re- duced when actors carry object of matching color. Cells denoted with stars are delivery locations. If an object is delivered to the location with the matching color, then its value is rewarded.
Σ SC
us [ASES
(ALa) (Y]T)M
ASES (A...) (Y],
where coefficients wy are set to wg
|s||nn- n!!
Next, we define a number of desirable properties for the attribution of the total agent-specific effect. These prop- erties are inspired from the game theory literature (Jain & Mahdian, 2007; Shoham & Leyton-Brown, 2008; Young, 1985) and translated to our setting.
Implementation. We adopt a Mtanner-cttorRRpootee system akin to (Dasgupta et al., 2023). Planner is imple- mented using pre-trained LLM and few-shot learning, to provide actors with instructions. More specifically, Planner can instruct actors to: examine a box, pickup an object and deliver that object to a specific destination. Furthermore, we assume an optimal Reporter whose task is to report to Planner the necessary information about the state of the environment. particular, Reporter provides information about the boxes' contents and which objects were picked up by the actors. Finally, the two actors are trained with deep RL to follow the Planner's instructions.
Efficiency: The total sum of agents' contribution scores is equal to tot-ASE.
Invariance: Agents who do not contribute to tot-ASE are assigned zero contribution score.
Symmetry: Agents who contribute equally to tot-ASE are assigned the same contribution score.
Contribution monotonicity: The contribution score as- signed to an agent depends only on its marginal contribu- tions to tot-ASE and monotonically so.
Setup, For our demonstration purposes, we consider the (factual) trajectory illustrated in Fig. 2a. We intervene on the pickup action of actor Ag forcing it to disobey Planner and choose the green object. The resulting counterfactual trajec- tory can be seen in Fig. 2a as well. Additional results from second experiment, where we intervene on the Planner's action, can be found in Appendix K.1
We formally state these properties in Appendix F. We now restate in the setting of agent-specific effects studied here an existing uniqueness result for Shapley value.
Theorem 5.3. (Young, 1985) ASE-SV is unique attribu- tion method for the total agent-specific effect that satisfies efficiency, invariance, symmetry and contribution mono- tonicity.
Counterfactual effects. To measure the total counterfac- tual effect in this scenario, we estimate the value of the total
6

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
TCFE
0.40
0.35
40 40
40
40
40
40
40
<ifell ASE
-55
Contributtinn Reb
0.30
40
+555B
0.20
0.15
0.10
0.05
-RarnerSE
100
200
300
0.00
(0-10)
11
12
13
34 (15-20)
Effect Value
Time-Stop
(a) Factual and Counterfactual Trajectory
(b) Counterfactual Effects
(c) r-SSE Attribution
Figure 2a depicts the actors' movements both the factual and counterfactual trajectory used in our experiments. Initially, both A₁ and A (represented by solid circles) are instructed pickup the pink object and deliver it the pink delivery location. In the counterfactual trajectory, A2 is forced to pickup the green object instead, prompting Planner to issue an alternative instruction for delivery to the green location. This intervention does not affect A 's behavior. A textual depiction of both trajectories is provided in Appendix K.I. Plot 2b shows the values of various counterfactual effects computed on the trajectory's is discounted total reward. The minus sign indicates that the negative of these values are plotted. Plot 2c shows the contribution ratios attributed to all state variables by r-SSE-ICC. Averages and standard errors are reported for 5 seeds.
reward collected in the counterfactual trajectory, and sub- tract from it the observed return. For the total agent-specific effect (Definition 3.1), need to isolate the effect of the intervention that propagates only through the agents (A1, Ag and Planner). Compared to TCFE, we thus estimate the return of the counterfactual trajectory in which the stochas- tic penalties are realized as if A2 carries the pink object. For the state-specific effect (Definition 3.2), we have isolate the effect of the intervention that propagates only through states. Therefore, we estimate the return of the counterfac- tual trajectory in which agents take their factual actions, but stochastic penalties are realized as if Ag carries the green object. For the reverse SSE (Eq. 2), it suffices to com- pute the difference between the returns the counterfactual trajectories considered for tot-ASE and TCFE.
the only sources of stochasticity in the environment. More- over, we observe that the scores attributed the four states decrease over time. Since penalties sampled indepen- dently, this can be interpreted as follows: the uncertainty over the counterfactual penalty estimates is greater in earlier time-steps. The latter can be confirmed by comparing the penalty distributions from Table 2 in Appendix J.
6.2. Sepsis
Environment. The two-agent variant of the sepsis treatment setting (Triantafyllou et al., 2024) we consider here involves clinician and an Al agent who take sequential actions in turn-based manner for treating an ICU patient. At each of 8 possible treatments,
round, the Al recommends which is then reviewed and potentially overridden by the clinician. The likelihood of the clinician overriding the Al's treatment at any given state is modeled by parameter which is varied in our experiment. Intuitively, μ serves as a proxy for the clinician's 's level of trust in the Al's recommen- dations: higher values of μ correspond greater levels of trust. If the Al's action is not overridden, then its selected treatment is applied. Otherwise, new treatment selected by the clinician is applied. The outcome of a trajectory is deemed successful if the patient is kept alive for 20 rounds or gets discharged earlier.
Causal explanation formula. Plot 2b indicates that TCFE is not decomposed into tot-ASE and SSE. Theorem 3.3, the other hand, is empirically validated in this scenario.
ASE-SV. According to Plot 2b, the ASE-SV attributes zero scores to both A1 and Planner, while assigning the full tot- ASE to A2. A lack of contribution to the effect is due to its unresponsiveness to A₂'s actions. Although the Plan- ner does respond A2, the environment's s state. As result, the effect of our inter- vention on the total reward does not propagate through the Planner's actions. These represent two distinct mechanisms by which agents can be excluded from contributing to the total agent-specific effect.
unable directly influence
Evaluation of ASE-SV. We generate 600 trajectories with unsuccessful outcomes. We then measure the total coun- terfactual effect of all possible alternative actions the final state of these trajectories and keep those that exhibit TCFE 2 0.8. Through that process, 8728 alternative ac- tions are selected for the evaluation of ASE-SV, For all se- lected actions, we compute their total agent-specifie effect, clinician-specffee effect and Al-specific effect. As expected,
r-SSE-ICC. Plot 2c shows that r-SSE-ICC pinpoints four state variables with non-zero contributions the reverse state-specific effect. As expected, the time-steps of these variables coincide with the time-steps at which A2 traverses the colored cells in the counterfactual trajectory, as these are

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
1.00
0.80
Gini Cleffiiientt
0.80
0.60
0.60
00:40
5SSE
SSE
0.20
0.00
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.00
Trust Parameter
Trust Parameter j/
Rounds after Intervention
(a) Decomposition: AI
(b) Decomposition: CL
(c) Gini Coefficient Distribution
Figure 3: Plots 3a and3b show the average percentage decomposition of -r-SSE and scores det and thi attributed by ASE-SV w.r.. TCFE, for interventions on the actions of Al and clinician, respectively, while varying trust parameter Plot 3c shows the Gini coefficient distribution over the scores attributed to state variables by the r-SSE-ICC method. The x-axis displays how many rounds after the considered intervention the trajectory terminates.
the sum of the individual effects does equal the total one, with discrepancies of up to 95%. In contrast, in our experiments, ASE-SV always attributes the effect efficiently to the clinician and AI, as supported by Theorem 5.3.
variables by the r-SSE-ICC method. We are interested in seeing how spread the scores across the states, i.e., if method attributes the effect equally or if it assigns larger scores to few states. To achieve this, we depict the Gini coefficient distribution (Gini, 1936) of these scores for vari- ous round differences in Plot 3c. Our results indicate that, independently of the trajectory size, r-SSE-ICC pinpoints for most trajectories only small subset of state variables with significant (intrinsic) contribution to the reverse state- specific effect. In practice, this means that for this setting actually need to infer the counterfactual values of only few key states in each trajectory in order to accurately estimate r-SSE. This is an interesting observation, as implies that given set of trajectories, r-SSE-ICC can reveal aspects of the underlying counterfactual distribution.
Plots 3a and 3b show the average percentage composition of the reverse state-specific effect and the agent scores at- tributed by ASE-SV w.r.t. the total counterfactual effect, for different trust levels. Plot 3a (resp. Plot 3b) considers the average over all selected Al (resp. clinician) actions. Results reveal that our method demonstrates trend similar to the one described in (Triantafyllou et al., 2024). In particular, the amount of tot-ASE attributed to the clinician (resp. Al) decreases (resp. increases) as the level of trust rises, eventu- ally reaching zero (resp. full) when the clinician completely trusts the Al's recommendations. This observation intu- itive, since the clinician expected contribute less to the effect as it acts more infrequently in the environment, while at the same time the AI is expected to contribute more as it assumes greater agency. Thus, we conclude that ASE- SV can efficiently attribute tot-ASE without sacrificing the conceptual power agent-specific effects.
7. Discussion
In this paper, we introduce causal explanation framework tailored to multi-agent MDPs. Specifically, we decompose the total counterfactual effect an agent's 's action by attribut- ing
agents' behavior or environment dynamics. Our experimental results demonstrate that our decomposition provides valuable insights into the distinct roles that agents and environment play in influencing the effect. To the best of our knowledge, this is the first work that looks into the problem of counterfactual effect decomposition in the con- text of multi multi-agent sequential decision making. While our findings are promising, there are several directions for future exploration, which we outline below.
Evaluation of r-SSE-ICC. We consider the same setup before and categorize all selected actions based the difference between the round that they were taken and the final round of their respective trajectory. For instance, if the action we consider was taken by the AI at the third round of a trajectory with rounds in total then the round difference for that action is 5. For our analysis, we maintain actions with round difference between and 10. For all selected actions, compute their reverse state-specific effect together with its variance. We keep those with absolute r-SSE > 0.1 and variance 2 0.01, which yields total of 437 alternative actions for the evaluation of r-SSE-ICC.
Computational complexity. The computational complex- ity of our decomposition approach mainly depends on the
*In measuring the Gini coefficient consider only the state variables that follow intervention, and correspond to time-steps at which it was the Al's turn to take action. These are the only states that can be attributed a non-zero contribution by r-SSE-ICC.
For each selected action and its reverse stati-specific effect, we compute the contribution scores assigned to all state
8

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
total number of agents and the length of the MMDP's time horizon. In our experiments, we use a relatively small num- ber of agents and horizon of a few dozen time-steps. We believe that many interesting multi-agent settings belong to this regime, e.g., human-Al collaboration. Nevertheless, there are settings in which computational complexity consid- erations can be important, and we see this as an interesting future research direction to explore. In Appendix I, we analyze the computational complexity of the ASE-SV and r-SSE-ICC methods, and discuss potential mitigation strate- gies for when the number of agents or the time horizon are prohibitively large. We also discuss how agents' capabilities can affect the computational complexity of our approach.
full responsibility. We see significant potential in combining our approach with existing works blame attribution and related concepts in accountable decision making, offering practical benefits across various multi-agent domains.
Impact Statement
This paper aims to advance the field of Machine Learning, partieularly in accountable multi-agent sequential decision- making. Our approach to decomposing counterfactual ef- fects can be integrated into existing causal tools for ret- rospectively analyzing decision-making failures, offering deeper insights into accountability (see Section 7 for an example on blame attribution). While our work has poten- tial societal implications, we do not find any that require specific emphasis here.
Causal assumptions. Making causal assumptions in order to enable counterfactual identifiability is quite common in the literature. There is plethora of works at the intersection of decision making and counterfactual reasoning that as- sumes exogeneity alongside causal properties, such as weak (Triantafyllou et al., 2024) or strong (Tsirtsis & Rodriguez, 2024) noise monotonicity, counterfactual stability (Oberst & Sontag, 2019), or access to the ground-truie causal model (Richens al., 2022). However, these assumptions are often violated in practice. Extending the applicability of our pro- posed approach to non-identifiable domains would therefore be of significant practical value. For instance, our effect decomposition approach could utilize bounds on counter- factual probabilities instead of relying on point estimates, albeit on the expense of efficiency. Such bounds can be ob- tained from observational data through partial identification methods (Manski, 1990; Zhang et al., 2022).
A Acknovledgements
This research was, in part, funded by the Deutsche Fosshungsgemmeinschattt (DFG, German Research Foun- dation) project number 467367360.
References
Avin, C., Shpitser, I., and Pearl, J. Identifiability of path- specific effects. In International Joint Conference on Artificial Intelligence, pp. 357-363, 2005.
Beckers, S., Chockler, H., and Halpern, J. Y. A causal anal- ysis of harm. Advances in Neural Information Processing Systems, pp. 2365-2376, 2022.
Applications to accountablc decision making. We deem the problem of decomposing counterfactual effects partic- ularly relevant for multi-agent decision-making settings where accountability is paramount. Our approach can be ap- plied in these settings, by integrating into existing causal tools for retrospectively analyzing decision makinn failures. For instance, consider methods for blame attribution in multi-agent systems (Halpern & Kleiman-Weines 201 8; Friedenberg & Halpern, 2019). Typically, these methods first identify the agents' actions that were critical to the out- come, i.e., those that, had they been different, would have likely prevented failure. Next, they assess the agents' epis- temic states, determining to what extent each agent could or should have predicted the consequences of acting differently. Our approach enhance these methods by offering more granular notion of blame. In the Sepsis scenario described in Section 1, for example, the clinician may be expected to predict how their actions directly affect the patient's 's state, but may not be expected to predict the Al's responses, espe- cially if they have never worked with the current version of the model before. According the output of our decompo- sition approach (Plot lb), the clinician would then receive 73.5% of the total blame for their action, rather than bearing
Boutilier, C. Planning, learning and coordination in mul- tiagent decision processes. In Conference on Theoreti- cal Aspects of Rationality and Knowledge, pp. 195-210, 1996.
Chiappa, S. Path-specific counterfactual fairness. In AAAI Conference on Artificial Intelligence, pp. 7801-7808, 2019.
Chockler, H. and Halpern, J. Y. Responsibility and blame: A modell approach. Journal of Artificial Intelli- gence Research, pp. 93-115, 2004.
Correa, J., Lee, S., and Bareinboim, E. Nested counterfac- tual identification from arbitrary surrogate experiments. Advances in Neural Information Processing Systems, pp. 6856-688 1356-686,,4 2021.
Daniel, R. M., De Stavola, B. L., Cousens, S. N., and Vansteelandt, S. Causal mediation analysis with mul- tiple mediators. Biometrics, pp. 1-14, 2015.
Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., and Fergus, R. Collaborating with

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
language models for embodied reasoning. arXiv preprint arXiv:23022000763, 2023.
Kusner, M. J., Loftus, J., Russell, C., and Silva, R. Coun- terfactual faimess. Advances in Neural Information Pro- cessing Systems, 30, 2017.
Friedenberg, M. and Halpern, J. Y. Blameworthiness in multi-agent settings. In AAAI Conference on Artificial Intelligence, pp. 525-532, 2019.
Lynn, L. A. Artificial intelligence systems for complex decision- maiiing in acute care medicine: A review. Pa- tient safety in Surgery, 2019.
Gini, C. On the measure of concentration with special reference to income and statistics. Colorado College Publication, General series, 1936.
Madumal, P., Miller, T., Sonenberg, L.,, and Vetere, F. Ex- plainable reinforcement learning through a causal lens. In AAAI Conference on Artificial Intelligence, pp. 2493- 2500, 2020.
Halpern, J. Y. Actual causality. MiT Press, 2016.
Halpern, J. Y. and Kleiman-Weiner, M. Towards formal def- initions of blameworthiness, intention, and moral respon- sibility. In AAAI Conference on Artificial Intelligence, 2018.
Manski, C. Nonparametric bounds on treatment effects. The American Economic Review, 1990.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M.. Fidjeland, A. K., Ostrovski, G., et al. Human-level con- trol through deep reinforcement learning. Nature, pp. 529-533, 2015.
Heskes, T., Sijben, E., Bucur, I. G., and Claassen, T. Causal shapley values: Exploiting causal knowledge explain individual predictions of complex models. Advances in Neural Information Processing Systems, pp. 4778-4789, 2020.
Oberst, M. and Sontag, D. Counterfactual off-policy eval- uation with gumbel-max structural causal models. In International Conference on Machine Learning, pp. 4881- 4890, 2019.
Hicks, R. and Tingley, D. Causal mediation analysis. The Stata Journal. pp. 605-619, 2011.
Huang, W. Zhang, L.,, and Wu, X. Achieving counterfactual fairness for causal bandit. AAA/ conference on artificial intelligence, pp. 6952-6959, 2022.
Pearl, J. Direct and indirect effects. In Conference on Un- certainty and Artificial Intelligence, pp. 411-420, 2001.
Imai, K., Keele, L., and Yamamoto, T. Identification, infer- ence and sensitivity analysis for causal mediation effects. Statistical Science, pp. 51-71, 2010.
Pearl, J. Causality. Cambridge University Press, 2009.
Richens, J., Beard, R., and Thompson, D. H. Counterfac- tual harm. Advances Neural Information Processing Systems, pp. 36350-36365, 2022.
Imai, K., Keele, L.., Tingley, D., and Yamamoto, T. Un- packing the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, pp. 765-789, 2011.
Shapley, L. S. A value for n-person games. Annals of Mathematics Studies, pp. 307-317, 1953.
Jain, K. and Mahdian, M. Cost sharing. Algorithmic game theory, pp. 385-410, 2007.
Shoham, Y. and Leyton-Brown, Brown, K. Multiagent systems: Al- gorithmic, game-theoretic, and logical foundations. Cam- bridge University Press, 2008.
Janzing, D., Blöbaum, P., Minorics, L., Faller, P., and Mas- takouri, Quantifying intrinsic causal contributions via structure preserving interventions. In International Conference Artificial Intelligence and Statistics, 2024.
Singal, R., Michailidis, G., and Ng, H. Flow-based attribu- tion in graphical models: recursive Shapley approach. International Conference on Machine Learning, pp. 9733-9743, 2021.
Jia, R., Dao, D., Wang, B., Hubis, A., Hynes, N., Gürel, N. M., Li, B., Zhang, C., Song, D., and Spanos, C. J. To- wards efficient data valuation based on the Shapley value. In International Conference on Artificial Intelligence and Statistics, pp. 1167-1176, 2019.
Steen, J.. Loeys. T., Moerkerke, B., and Vansteelandt, S. Flexible mediation analysis with multiple mediators. American Journal of Epidemiology, pp. 184-193, 2017.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Jung, Y., Kasiviswanathan, S., Tian, J., Janzing, D., Blöbaum, P., and Bareinboim, E. On measuring causal contributions via do-interventions. In International Con- ference on Machine Learning, pp. 10576, 2022.
10

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Triantafyllou, S., Singla, A., and Radanovic, G. Ac- tual causality and responsibility attribution in decentral- ized partially observable Markov decision processes. In AAAI/ACM Conference on Al, Ethics, and Society, pp. 739-752, 2022.
Triantafyllou, S., Sukovic, A., Mandal, D., and Radanovic, G. Agent-specific effects: A causal effect propagation analysis in multi-agent MDPs. In International Confer- ence on Machine Learning, 2024.
Tsirtsis, S. and Rodriguez, M. Finding counterfactually optimal action sequences in continuous state spaces. Ad- vances in Neural Information Processing Systems, 2024.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double Q-learning. In AAAI confer- on artificial intelligence, pp. 2094 2100, 2016.
VanderWeele, T. and Vansteelandt, Mediation analysis with multiple mediators. Epidemiologic methods, pp. 95-115, 2014.
VanderWeele, T. Explanation in causal inference: de- velopments in mediation and interaction. International journal of epidemiology, pp. 1904-1908, 2016.
Wang, J., Wiens, J., and Lundberg, S. Shapley flow: A graph-based approach to interpreting model predictions. In International Conference on Artificial Intelligence and Statistics, pp. 721-729, 2021.
Young, H. P. Monotonic solutions of cooperative games. International Journal of Game Theory, pp. 65-72, 1985.
Zhang, J. and Bareinboim, E. Fairness in decision- making- the causal explanation formula. In AAAI Con- ference Artificial Intelligence, 2018a.
Zhang, J. and Bareinboim, E. Non-parametric path analysis in structural causal models. In Conference Uncertainty in Artificial Intelligence, 2018b.
Zhang. J., Tian, J., and Bareinboim, E. Partial counterfactual identification from observational and experimental data. In International Conference Machine Learning, pp- 26548-266 26558, 2022.

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
A. List of Appendices In this section, we provide a brief description of the content provided the appendices of the paper.
Appendix contains table that summarizes the most important notation used in the paper.
Appendix provides additional information on MMDP-SCMs.
Appendix D contains the causal graph of the MMDP-SCM from Section 2.2.
Appendix E provides additional information on noise monotonicity.
Appendix F formally states the properties defined in Section 5 for the ASE-SV method.
Appendix G outlines an algorithm for approximating the conditional variance from Eq. 4.
Appendix contains the proofs of Theorems 3.3 and 4.2.
Appendix provides a discussion on the computational complexity of the ASE-SV and r-SSE-ICC methods.
Appendix J provides additional information on the experimental setup and implementation details.
Appendix K includes additional experimental results.
Appendix L includes additional experiments assessing the estimation enror of our empirical results.
Appendix M includes additional experiments assessing the robustness of our empirical results to the noise monotonicity assumption.
Appendix N provides a graphical illustration of all counterfactual effects introduced in Sections and using the Sepsis example from (Triantafyllou al., 2024).
12

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
B. Notation Summarization Table
Table 1: Summarizes the most important notation used in the paper.
Notation
Meaning
M
MMDP-SCM
P(-M
Probability defined over M
(1,.,, n
Set of agents
Time horizon
St, 8t. E S
State variable and value at time-step
As, ai, Ai
Action variable and value of agent at time-step
T()
Trajectory and value of variable X in Probability conditioned on trajectory generated by M'
P(-[T; MM)M
U,
Vector of noise variables and vector of noise values
P(u), P(ulT)
Prior and posterior noise distributions
dol Ac, (i.r)
Hard intervention Ag.
Ar,
Modified MMDP-SCM
Response/Outcome variable
Ya,.
Potential response of Y [o(A,,t == ai,1)
do Yar,))
Natural intervention Y
P(ya,. T)
Counterfactual probability of Y = y under do(A := ai.))
TCFE (Ac,)) (X
Definition 2.1: Total counterfactual effect (TCFE)
of interventions on action variables
ASE (Y(T)
Definition 3.1: Total agent-specific effect (tot-ASE)
SSE (1) (Y
Definition 3.2: State-specific effect (SSE)
SSE,(AL (A:.)aa (Y
Equation 2: Reverse state specifi effect (r-SSE)
AYi.a ICC(S& - AY.....
Difference in potential responses Y, Yau.
Equation 4: Intrinsic causal contribution (ICC)
Score assigned to state S₂ by the r-SSE-ICC (Definition 4.1)
ASEN (A,,) (Y(T)M
Definition 5.1: Agent-specific effect (ASH)
Øi
Score assigned to agent by the ASE-SV (Definition 5.2)
C. Additional Information on MMDP-SCMs
Consider an MMDP-SCM M UV, P(u). For the observational distribution of M, P(V), to be consistent with an MMDP (S, (1,(11,.,)) (1,...,,), m), A, T, h, a) and joint policy æ, functions in F and noise distribution P(u) need to satisfy the following conditions for every (s, a, triplet and time-step t:
Junoospottto) P(wSo) = PPSo-=sof s|o); J P(u ((s'|ss,))
J P(rd.1)= = (a)(s)))
(5)
The first two conditions in Eq. 5 guarantee that M induces the initial state distribution and state transition dynamics of the MMDP. The third condition makes sure that the action variables in M agree with the joint policy æ
D. Causal Graph of MMDP-SCM
This section contains the causal graph the MMDP-SCM described in Section 2.2. The causal graph shown in Fig. 4.
13

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Aba
...
Anua
Auc
Aua
Aix-t
Aua-t
So
S6-1
%
Figure 4: The causal graph of an MMDP-SCM with n agents and horizon h. Exogenous variables are omitted.
E. Additional Information on Noise Monotonicity
In this section, we define the (weak) noise monotonicity property for categorical SCMs. has been shown that noise monotonicity enables counterfactual identifiability. For more details on noise monotonicity and its connection to the identifiability problem, refer the interested reader (Triantafyllou al., 2024).
Definition E.1 (Noise Monotonicity). Given an SCM M with causal graph G, we say that variable V E V is noise- monotonic in M w.r.t. a total ordering Si on dom V*), if for any pa' E domm{P' (Pa (G)} and u wz ~ P(U)) s.t. ui < uzz holds that ('(pa,, uf) Si f (ppa', uz))
Essentially, noise monotonicity assumes that all observed variables in an SCM, or MMDP-SCM in our paper, are monotonic w.r.t. their corresponding noise variable (for some specified total ordering). Note that noise monotonicity is not limiting for the MMDPs or agents' policies. In simple words, what noise monotonicity assumption restricts the expressivity of counterfactual distributions. There can many MMDP-SCMs whose observational distribution consistent with the MMDP, but admit different counterfactual distributions. Theorem 4.3 in (Triantafyllou et al., 2024) shows that by limiting the class of possible MMDP-SCMs to the that satisfy noise monotonicity, counterfactual identifiability is guaranteed.
F. Properties for ASE-SV
In this section, we formally state the properties defined in Section for the ASE-SV method.
Efficiency: The total sum of agents' contribution scores equal to the total agent-specific effect, Formally,
Σ
Φj ASE(I....)) (ALe)(ALe) (Y]T)M-
(Y
Invariance: Agents who do not marginally contribute to the total agent-specific effect are assigned zero contribution score. Formally, if for every S S n.V{)
ASE
(A,,) (Y(7)M ASES ASE T (Y(T)M -0,
then dy
Symmetry: Agents who contribute equally the total agent-specific effect assigned the same contribution score. Formally, if for every 8 C (1,...,,t)) (1....,n]\(i.k)
ASE
SU(i) (A,.) (Y|7)
ASE
SUE)) ( (
ASES
then dy =0 Øk-
Contribution monotonicity: The contribution score assigned an agent depends only on its marginal contributions the total apent-specific effect and monotonically so. Formally, let Mi and M2 be MMDP-SCMs with n agents, for every S S (1.
m
SUH (A,. (Y(7)M, ASE a (A.,)) (Y(7)M,
ASESUD) (Y(7) (A.))(A..))
ASES ... (At.)) (Yl7))
then 03 2 2 Mo
14

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
G. Algorithm for Conditional Variance
In this section, we present our approach for approximating the expected conditional variance from Eq. Algorithm I estimates E[ IT, U []]M.. To estimate the conditional variance E Va U [ suffices modify Algorithm sampling conditioning noise variables from P(u (8,, u (us1r] [r) and non-conditioning ones from ((u2Sk-11T).
Algorithm Estimates B| |T, U /|
Input: MMDP-SCM M, trajectory 7, action variable Ai,,, action ai,,, response variable Y, state variable Sg, number of condttoniigghonnccodttinnnngg posterior samples Hz/H₂
1: h1 < 0, h2 0
2: in < 0, 1/2 11 do
3: while
4: Ucond (T)
# Sample conditioning noise variables
5: h - h1 +1
6: 0,,000 4-
7: while /12 < H2 do
8:
Usen ~ P(u228(T)
# Sample non-conditioning noise variables
9:
h22 + h22 +1 +1
(Ulond, Umon)
aat P(V[u) M modo) Compute counterfactual trajectory (V)
of
I [Ap = (Av, ())
14:
y' P(Y[u) Mares
# Compute response to natural intervention
15:
+ (y' -)a)
16:
t + (y' y0
17:
end while
18:
H1 - 11 + (1))) )
1-t-12+ Ha
h260
21:
end while return #2ー
22:
H. Proofs
H.1. Proof of Theorem 3.3
Proof. Eq. 3 follows directly from Definition 2.1, Definition 3.1 and Eq. 2:
TCFE (A4,, = EE...,, IT|M T(Y)
1.Y...77J.. 7(Y) + [T M MMaann -E Ir; M MMaan
ASE((....))
(p (Y - SSE (Y
where I = (Ar.v == Ar Aest
15

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
H.2. Proof of Theorem 4.2
Proof. It holds that
E[Var(AY, 1,,, U<$++1) |
E Var( (A 1,a,,,12 Us U }rr (SE,(AL)),.,, (YT)M
US
Var ( V (T)
+E[[+1,t
Vaar(AVV.oo |T, U<8111) ) 10--BR] [a AVI 17, U)/T|M (YE
Var( Bar(AV) Bar(AY1,ai..17
B[Var(AY,,o, [T, UK 1 SSE (YIT)
Var(AYr.oo
First step holds because E|[ar( G[Var(AYY,,ar AYT, as [T, U U = E[Var(AY), U< [r]] M. for every k E {t +
1.... ty - 1.. The second step follows from the fact that noise termss associated with observed variables which (chronologi- cally) proceed by do not influence the value of AVI,a,,- The third step holds because the expected conditional variance satisfies calibration, i.e., [[Var(AYr,,o,,
U]| O.
Let now X be any ancestor of S1+1 in the causal graph of M, apart from Ar,. Note that is not affected by interventions do(T) and do( A,,; d. Therefore, the solution of X in the MMDP-SCMs Mdo(T) and Mdo(AA..=o..)) will be equal to its factual value in i.e, (X), for every context sampled from the posterior P(ulr). Furthermore, A, is fixed to ag in M (dd(A.. while it is also not affected by d). It follows that conditioning on the noise terms associated with X or A does not reduce the variance of AYi, Therefore, it holds that
E Var(AY U Jr]]as = Var(AY
which concludes our proof.
I. Discussion on Computational Complexity
In this section, we analyze the computational complexity of the ASE-SV (Definition 5.2) and r-SSE-ICC (Definition 4.1) methods, and discuss potential mitigation strategies for when the number of agents the length of the time horizon prohibitively large. We conclude the section with a discussion about the effect of agents' capabilities on the computational complexity of our approach.
L1. Computational Complexity of ASE-SV
The number of agent-specific effect evaluations required by the exact ASE-SV calculation grows exponentially with the number of agents One potential mitigation strategy for this problem is to adapt setting sampling based approaches that efficiently approximate Shapley value without violating efficiency, i.c., attributing the entire effect. (Jia et al., 2019) propose such an algorithmic approach, which requires ([n(log n) evaluations for any bounded utility. This means that their algorithm is applicable to ASE-SV in settings where the value of agent-specific effects is bounded, as is the case in both our experiments.
L.2. Computational Complexity of r-SSE-ICC
Computing the contribution scores assigned by the r-SSE-ICC method to all state variables requires O(h), where h denotes time horizon, executions of Algorithm When we deal with long-horizon MMDPs, this linear dependence on the number time-steps can slow down our method. One intuitive strategy reduce the number of computations in this case by grouping together state variables from consecutive time-steps. That way, the r-SSE-ICC method would attribute the effect to sets of consecutive state variables instead of individual ones. If the time horizon (between action and outcome) is partitioned in groups of the same fixed size k. except maybe for the last one, then the modified r-SSE-ICC method would require ( executions of Algorithm 1.
In settings where it is reasonable to assume or empirically verify that r-SSE-ICC is sparse, in the sense that only a few state variables have significant (intrinsic) contributions the effect, as the case both our experiments,
16

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
then we are able further reduce the number of Algorithm 1 executions. More specifically, we can utilize the fact that the expected noise-conditional variance measure satisfies monotonicity, i.e., ElVar(A (iVa,,AYr,a,, l7, J= 2 E[Var( (iVar(AY,nnn Aa,,,, U< |T] If we know, for example, that for most of the times there is at most one state variable with non negligigeee (intrinsic) contribution to the effect, we can simply use binary search approach to pinpoint that state. This can reduce the complexity of r-SSE-ICC to O(log(h)) executions.
L.3. Effect of Agents' Capabilities on Computational Complexity
The complexity of decision decision-making agents affects the computational complexity of our decomposition approach, assuming that increased capabilities imply increased inference time. The reasoning is the following: our approach to estimating counterfactual effects involves sampling trajectories from the posterior distribution and then averaging the values of the response variable across these trajectories. Sampling a trajectory from the posterior distribution generally requires to prompt each agent once for every counterfactual state which they need act.
For reference, in the Gridworld environment, more than 90% of the time required to sample one counterfactual trajectory is spent on the inference of the LLM agent, while the remaining ~ 10% is shared between the two RL agents.² 2 Consequently, if we were to use an LLM agent with reduced cognitive capabilities, and hence less inference time, then the scalability of our approach in this experiment would significantly improve.
J. Experimental Setup and Implementation
In this section, we provide additional information the experimental setup and implementation.
J.1. Gridworld
Setup. Our setup is an adaptation of the Wlanner-Actor-Reoorte system from (Dasgupta et al., 2023). The Planner is tasked with understanding the high-level steps necessary for the completion of a task and then breaking it down to sequence of instructions. Actors are RL agents pre-trained to complete a set of simple instructions in the environment. Lastly, the Reporter is tasked with translating environment observations into a textual representation comprehensible by the Planner.
Environment. We consider the gridworld environment depicted in Fig. 2a with two actors, Ai and A2. There are two boxes located on the rightmost comers, each of which contains two objects. Each object has a color that determines its value, particular, pink > green > yellow. The object's color is randomly sampled at the beginning of each trajectory. Objects can be picked up and carried by the actors each actor can pick up only one object, and only one object can be picked up from each box. Grey-colored cells represent walls. Blank cells indicate areas of small negative cost. Colored cells indicate areas of larger stochastic penalty, which is significantly reduced when actors carry an object of a matching color. Penalties induced by cells of the same color share the same means, but might differ in their underlying distributions. Moreover, in expectation, pink cells inflict higher penalties than green ones, and green cells higher than yellow ones. Cells denoted with stars are delivery locations. If an object is delivered to the location with the matching color, then the object's ss value is rewarded. The objective in this environment is to maximize the combined total return of both actors. The full reward specification can be found in Table 2.
Instructions. The Gridworld environment supports simplified set of 8 instructions: examine box 1, examine box 2, pickup pink, pickup green, pickup yellow, goto pink, goto green and, goto yellow. We pre-train both actors to learn coal-conditioned policy for executing each of the available instructions. During training, we sample new instruction at the beginning of each trajectory. Additionally, we initialize an actor according to the instruction and randomize over its valid observation space. For examplc, for the instruction goto pink, we initialize the actor to its respective position (under/above the first/second box for A₁ and A respectively) and randomly select the object it's carrying. The actor is rewarded positively whenever it completes the instruction.
Actors. Actors A₁ and Ag spawn on the same fixed locations at the beginning of cach trajectory. Apart from movement actions, actors can also perform pickup actions when located next to a box. The policies are represented via neural network parameters and are learned using double deep Q-learning (Mnih et al., 2015; Van Hasselt et al., 2016). Both agents take as their input concatenated, one-hot encoded vectors, which include their instruction, their current position and the color of the
'Raw values and additional details on the time compute of our experiments are included on the README file of our code. which can be found at https://github.co
git.
17

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Table 2: Reward specification for Gridworld. An empty distribution column implies a deterministic reward issued upon entering the cell. For the green corridor, penalties are specified on a per-cell basis, identified by their zero-based indices into the associated row and column.
Cell
Values
Distribution
All
-0.2
Pink Penalty
[-30, -50,
[1/3, 1/3, 1/3]
Pink Penalty (Reduced)
[-5,-15, -25]
[1/3, 1/3, 1/3]
Pink Delivery
+180
Green Penalty C2
[-30, 40,-500
[0.3, 0.4, 0.3]
Green Penalty C2 (Reduced)
-50,-001155
[0.3, 0.4, 0.3]
Green Penalty C2
-40,
[0.25, 0.5, 0.25)
Green Penalty C2.5 (Reduced)
[-5,-10,-15]
[0.25, 0.5, 0.25)
Green Penalty C2
-40,
[0.2, 0.6, 0.2]
Green Penalty C2 (Reduced)
[-5,-10,-15]
[0.2, 0.6, 0.2]
Green Penalty C2,7
-30,-00-550)
[0.15, 0.7, 0.15)
Green Penalty C2,7 (Reduced)
[-5,-10,-15]
[0.15, 0.7, 0.15)
Green Delivery
+150
Yellow Penalty
-30,-30,
[1/3, 1/3, 1/3]
Yellow Penalty (Reduced)
-2,
[1/3, 1/3, 1/3]
Yellow Delivery
+90
object they are carrying. We provide a full list of hyperparameters in Table 3. The hyperparameter optimization method was performed by randomly sampling 50 candidates from the specified ranges and selecting the combination that yielded the best test reward, averaged over instructions.
Planner and Reporter. Planner is implemented using pre-trained LLama 2.7B model (Touvron al., 2023) and few-shot learning, to provide actors with instructions. More specifically, Planner instruct actors examine box, pickup an object and deliver that object to specific destination. Furthermore, we assume an optimal Reporter whose task is to report to the Planner the necessary information about the state of the environment. In particular, the Reporter provides information about the boxes' contents and which objects were picked up by the actors (see Trajectories and for an illustrative example).
J.2. Sepsis
Our experimental setup and implementation closely follow that of (Triantafyllou et al., 2024).
J.3. Compute Resource
All experiments were run on a 64bit Sabian-based machine having 2x12 CPU cores clocked at 3GHz with access to TB of DDR3 1600MHz RAM and an NVIDIA A40 GPU. The software stack relied on Python 3.9.13, with installed standard scientific packages for numeric calculations and visualization (we provide a full list of dependencies and their exact versions as part of our code).
18

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Table 3: Hyperparameters used for the Gridworld actors' policies.
Parameter name
Parameter value
Tuning Range
Discount
0.99
[0.99, 0.9, 0.8]
Target Update Freq.
1000
[500, 1000, 1500]
Batch size
512
[256, 512, 1024, 2048]
Hidden Dim
128
[64, 128, 256]
Hidden Depth
[2 [le-5, 5e-5, le-4, 5e-4, le-3]
Learning Rate
le-4
Num. Estimation Step
[1, 10, 15]
total ASE
ASSE
.6-SF
Panner-S
(a) Counterfactual Effeets
Figure 5: Gridworld: Plot 5a shows the values of various counterfactual effects computed on the trajectory's s total collected reward for the case when we intervene the Planner's action at Step 2, forcing it instruct Ag to pick up the green object instead of the pink one. Averages and standard errors reported for different seeds.
K. Additional Experimental Results
K.1. Gridworld
Additional experiment. We repeat the experiment from Section 6.1, but instead of intervening on A pickup action intervene on the Planner's action. In particular, we intervene the Planner's 's action Step 2, forcing it to instruct A to pick up the green object instead of the pink one. The total counterfactual effect of this intervention is equal to that of the intervention on A2's 's action. However, the result of our decomposition approach for these two effects is different.
According to Plot 5a, the TCFE in this scenario is fully attributed to how the agents would respond to the intervention, and more specifically the response of agent Ag. Both the SSE and the r-SSE in this scenario are zero. This result intuitive, the Planner is able to influence the state transitions directly, and hence the effect of its actions do propagate through the environment dynamics. In contrast, the actions of Ag can influence the outcome through both the environment and future agent actions, and hence the decomposition of their effect is more nuanced (see Plot 2b).
Trajectories. We provide textual depiction of the factual (Trajectory 1) and counterfactual (Trajectory 2) trajectories from Fig. 2a. We also provide a textual depiction of the counterfactual trajectory from the experiment described above (Trajectory 3).
K.2. Sepsis
Fig. 6 illusstrtess the distribution of the r-SSE contribution scores computed by the r-SSE-ICC method in Section 6.2. Due to the use a finite number of samples for estimating these scores, some negative values may be erroneously attributed. These cases have been excluded from the plots.
19

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Gridworld Trajectory 1 Factual
Box 1: (PINK, YELLOW); Box 2: (PINK, GREEN)
Step: 0; Reporter: Aj respawn; As respawn;
Planner: (examine box 1, examine box 2); Reward 0.0;
Step: Actors (A1, A2): up, down; Reward: -0.4 (A1: -0.2, A2: -0.2):
Step: 2; Reporter: A₁ (PINK YELLOW); A2 (PINK GREEN);
Planner: (pickup pink, pickup pink); Reward 0.0;
Step: 3; Actors (Ai, A2): pickup pink, pickup pink; Reward: -0.4 (Ai: -0.2, A2: -0.2);
Step: Reporter: A₁ has PINK; A2 has PINK; (goto pink, goto pink); Reward 0.0;
Planner:
Step: 5; Actors (A1, A2): down, up; Reward: -0.4 -0.2, -0.2);
Step: 6; Actors (A1, A2): left, up: Reward: -0.4 (A₃: -0.2, -0.))
Step: 7; Actors (Ai, A2): left, up: Reward: -0.4 (Aι: -2 -0.)):
Step: Actors (A1, A2): left, left; Reward: -0.4 (A1: -0.2, -02)):
Step: Actors (A₁, A2): left, left; Reward: -0.4 (A,: -b. -0.2);
Step: 10; Actors (A1, Ag): left, left; Reward: -5.4 (A1: -25.2, -0.2);
Step: 11; Actors (A1, A2): left, left; Reward: -254 (A₁: -2..2,
42: -0.2)
Step: 12; Actors (A,, A2): left, left; Reward: -40.4 (A₁: -15.2, -25.2);
Step: 13; Actors (A1, Ag): left, left; Reward: -3044 (AІ: -5.2, -25.2;;
Step: 14; Actors (А₁, A2): up, left; Reward: -2..4 -0.2, Az: -2.2);
Step: 15; Actors (A1, A2): left, left; Reward: -254 (Aς: -0.2, -2522):
Step: 16; Actors (A,, Ag): left, left; Reward: -A, -0.2, Ag: -0.2;
Step: 17; Actors (A1, Ag): left, up; Reward: -0.4 (Aς: -4.2,
-02);
Step: 18; Actors (A1, A2): NULL,, left; Reward: -0.4 (A1: -0.2, 42: -0.2):
Step: 19; Actors (A1, Ag): NULL, left; Reward: -0.4 (A1: -022 Ag: -0.2):
Step: 20; Goal Reward: 360.0; Total Reward: 183.2;
20

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Gridworld Trajectory 2: Counterfactual (Ag² 's action)
Box 1: (PINK, YELLOW); Box 2: (PINK, GREEN)
Step: Reporter: A₁ respawn; 42 respawn:
Planner:
(examine box 1, examine box 2); Reward 0.0;
Step: 1; Actors (Ai, A2): up, down; Reward: -0.4 (A1: -0.2, A2: -0.2);
Step: 2; Reporter: A (PINK YELLOW); A2 (PINK GREEN);
Planner: (pickup pink, pickup pink); Reward 0.0;
Step: Actors (A1, A2): pickup pink, pickup green; Reward: -0.4 (A1: -0.2, A2: -0.2);
Step: 4; Reporter: A₁ has PINK; A2 has GREEN;
Planner: (goto pink, goto green); Reward 0.0;
Step: 5; Actors (A1, A2): down, up: Reward: -0.4 (A1: -0.2, A2: -0.2):
Step: Actors (Ai, A2): left, left; Reward: -0.4 (A₁: -0.2, A:: -0.2);
Step: Actors (A1, A2): left, left; Reward: -0.4 (A₁: -0.2, -02)):
Step: Actors A2): left, up: Reward: 0.4 (A₁: Ag. -0.2):
Step: Actors (A1, A2): left, left; Reward: -0.4 (A1: -0.2, -0.2);
Step: 10; Actors (A1, A2): left, left; Reward: -2.4 (AІ: -25.2,
42: -0.2):
Step: 11; Actors (A1, A2): left, left; Reward: -35.4 (A₁: -25.2, -10.));
Step: 12; Actors (A1, Ag): left, left; Reward: -30.4 (AІ: -15.5,
-15.2);
Step: 13; Actors (А₁, A2): left, left; Reward: -20.4 (AІ: -5.2,
Ag: -15.2);
Step: 14; Actors (A1, A2): up, left; Reward: -15.4 -15.4AAr -0.2, A2: -15.2);
Step: 15; Actors (A,, Ag): leftt left; Reward: -0.4 (Ai: -0.2,
A2: 0.2);
Step: 16; Actors (A1, Ag): left, left; Reward: -0.4 (AІ: -0.2, -0.2);
Step: 17; Actors (A,, A2): left, left; Reward: 0..4 (A1: -0.2, A2: -0.2);
Step: 18; Goal Reward: 330.0; Total Reward: 199.0;
21

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Gridworld Trajectory 3 Counterfactual (Planner's action)
Box 1: (PINK, YELLOW); Box 2: (PINK, GREEN)
Step: Reporter: A₁ respawn; 42 respawn:
Planner:
(examine box 1, examine box 2); Reward 0.0;
Step: 1; Actors (Ai, A2): up, down; Reward: -0.4 (A1: -0.2, A2: -0.2);
Step: 2; Reporter: A (PINK YELLOW); A2 (PINK GREEN);
Planner: (pickup pink, pickup green); Reward 0.0;
Step: Actors (A1, A2): pickup pink, pickup green; Reward: -0.4 (A1: -0.2, A2: -0.2);
Step: 4; Reporter: A₁ has PINK; A2 has GREEN;
Planner: (goto pink, goto green); Reward 0.0;
Step: 5; Actors (A1, A2): down, up: Reward: -0.4 (A1: -0.2, A2: -0.2):
Step: Actors (Ai, A2): left, left; Reward: -0.4 (A₁: -0.2, A:: -0.2);
Step: Actors (A1, A2): left, left; Reward: -0.4 (A₁: -0.2, -02)):
Step: Actors A2): left, up: Reward: 0.4 (A₁: Ag. -0.2):
Step: Actors (A1, A2): left, left; Reward: -0.4 (A1: -0.2, -0.2);
Step: 10; Actors (A1, A2): left, left; Reward: -5.4 (A₁: -2..2, 42: -0.2):
Step: 11; Actors (A1, A2): left, left; Reward: 35.4(Ai; -2..2, -10.2);
Step: 12; Actors (A1, Ag): left, left; Reward: -25.4 (AІ: -15.2,
-10.2);
Step: 13; Actors (А₁, A2): left, left; Reward: -15.4 (AІ: -5.2,
-10.2);
Step: 14; Actors (A1, A2): up, left; Reward: -1.4 -0.2, A2: -15.2);
Step: 15; Actors (A,, Ag): leftt left; Reward: -0.4 -0.2,
A2: -0.);
Step: 16; Actors (A1, Ag): left, left; Reward: -0.4 -0.2,
-0.2);
Step: 17; Actors (A,, A2): left, left; Reward: 00.4 (A1: -0.2, A2: -0.2);
Step: 18; Goal Reward: 330.0; Total Reward: 209.0:
22

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
Contribution Ratio
Contribution Ratio
Contribution Ratio
Contribution
2.0
0.3
0.00
030
2.30
0.00
0.00
3.00
3.00
Round Intervention
Round after Intervention
Intervention
Intervention
Round Difference
(b) Round Difference
(c) Round Difference
(d) Round Difference
Contribution Ratio
0.43
8-43
Contribution Ratio
0.43
0:30
Contribbution
8:30
0:30
0.23
0.23
Round after Intervention
Round after Intervention
Round after Intervention
(e) Round Difference 8
(f) Round Difference
(g) Round Difference 10
Figure 6: Sepsis: Plots 6a - 6g show the average contribution ratios attributed to the different state variables by the r-SSE-ICC method. Results are grouped based on the round difference of the selected actions (see Section 6.2 for an explanation).
plot the contributions only for state variables corresponding to rounds that follow the intervention All other contributions are zero. Averages and standard errors are reported for the 437 alternative actions chosen for the evaluation of r-SSE-ICC following the process described in Section 6.2.
23

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
0.40
TCFE
10tal ASE
total ASE
Contributiinn Ratio
0.30
0.25
S5E
6SE
+155E
SSE
0.16
SE
0.10
SE
-E
-Paamer-SE
-Plannr-SEE
0.00
0
100
200
300
10
15
20
25
[0-10]
[15-20]
Effect Value
Effect Value
Time-Step
(a) Counterfactual Effects: A2
(b) Counterfactual Effects: LLM
(c) r-SSE Attribution
Figure 7: Plots 7a, 7b, and 7c replicate the empirical results from Plots 2b, 5a, and 2c, respectively, while additionally including the corresponding ground-truth values for all quantities. Estimated values from the original plots are shown in blue, while the ground-truth values are depicted in green.
L. Additional Experiments Evaluating Estimation Error
To approximate counterfactual effects across all experiments presented in Section 6 and Appendix K, we employ posterior sampling-bane methods akin to Algorithm 1. This is a standard approach to counterfactual inference ((Pearl, 2009)). In this section, we present additional experiments to evaluate the estimation error and support the reliability of our empirical findings.
L.1. Gridworld
Fig. reproduces Plots 2b, 2c and 5a, now including the ground-truth values of all estimated quantities for comparison. Notably, the ground-truth values (green) consistently lie within the standard error bounds of the estimated quantities (blue). The ground-truth counterfactual distribution for this experiment was obtained through direct computation.
L.2. Sepsis
Compared to the Gridworld environment, acquiring ground-truth values for counterfactual quantities in the Sepsis setting significantly more challenging. Instead, analyze the standard error distributions by repeating the experiment across 10 different seeds. Specifically, for each alternative action selected for the evaluation of ASE-SV and r-SSE-ICC in Section 6.2, we perform the evaluation process 10 times and compute the empirical standard error for all estimated quantities: TCFF, tot-ASE, SSE, r-SSE, det, Pai. and r-SSE-ICC. Fig. 8 illustrates the resulting standard error distributions."
The plots from Fig. 8 reveal minimal variability in the estimates of our causal explanation formula across seeds, with only a very small number of outliers. These results reinforce the reliability of our findings from Section 6.2 and support the robustness of our effect decomposition approach in the Sepsis experiment.
M. Additional Experiments Evaluating Robustness to Noise Monotonicity
Throughout all experiments in this paper, we assume that noise monotonicity holds (see Appendix E for formal definition) w.r.t a chosen set of total orderings. In the Gridworld experiment, we design the environment such that penalty variables are noise-monotonic w.r.t. the numerical ordering - all other variables this experiment are deterministic. In the Sepsis experi- ment, however, we lack access to the underlying causal model and rely solely on observational distributions. Consequently, the choice of total orderings for noise monotonicity in this experiment may influence the results. this section, we present additional experiments evaluate the robustness of the empirical findings from Section 6.2 to variations in the choice of the total orderings. We note that similar evaluation conducted in (Triantafylloou et al., 2024).
"We chose plot standard error distributions grouped by the absolute average means of their estimates other metrics of relative dispersion, such as Coefficient Variation, due to the fact that for many our estimates their mean value centers close to zero.
24

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
0.06
Standard Error
Standard Error
Sandaad Error
002
0.02
0.02
0.01
0.01
0.01
0.01
(0. -2] 12, 4] (4.66 18. Al 18.1]
(4.6)
(8.1)
4l
(8.
[0.-2]
(A.
Absolute Mean Range
Absolute Mean Range
Absolute Range
Absolute Range
(a) TCFE
(b) Total ASE
(c) SSB
(d) r-SSE
0.05
0.06
0.30
0.04
0.34
0.25
Standard Error
Stendard Error
0.20
0.m8
6.08
Stenderd
0.15
0.02
(2..4)
Absolute Mean Range
Absolute Mean Range
Absolute Mean Range
(c) Oas
(f) del
(g) P---SE-ICC (max)
Figure 8: Box Plots 8a-8f show the standard error distributions of all counterfactual estimates from Section 6.2 over all 8728 alternative actions selected for the evaluation of ASE-SV in that section. Box Plot 8g shows the standard error distribution of the scores assigned by the r-SSE-ICC method for a similar set of alternative actions as the one used in Section 6.2 for the evaluation of r-SSE-ICC. Among the multiple standard errors associated with each alternative action (one for each score assigned a state variable), report the one with the largest value. Standard errors and absolute mean values are measured across 10 different seeds.
25

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
1.00
1.00
0.80
Gini Coefficient
0.60
0.00
0.40
1SSE
SSEE
0.20
0.00
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.6
Trust Parameter
Trust Parameter j/
Rounds after Intervention
(a) Decomp. Al: Ordl
(b) Decomp. CL: Ordl
(c) Gini Coefficient: Ordl
1.00
Ginil Coefficient
0.80
TCR6
0.60
0.60
0.60
0.40
0.40
00.40
55S
SSE
0.20
0.20
0.20
0.00
0.00
0.00
Trust
Parameter
Rounds after Intervention
(d) Decomp. Al: Ord2
(c) Decomp, CL: Ord2
(f) Gini Coefficient: Ord2
1.00
1.00
0.80
0.80
Gini Coefficient
0.80
0.60
0.60
0.40
0.40
0.40
KSSE
0.20
0.20
0.00
8.4
0.00
0.00
6
Trust Parameter y
Trust Parameter μ
Rounds after Intervention
(g)
Decomp. Al: Ord3
(h) Decomp. CL: Ord3
(i) Gini Coefficient: Ord3
1.00
1.00
1.00
0.80
0.80
Gini Coefficient
0.80
TCFe
0.60
0.60
0.60
0.40
0.40
0.20
0.20
0.00
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
10
Trust Parameter
Trust Parameter p
Rounds after Intervention
( Decomp. Al: Ord4
(k) Decomp. CL: Ord4
(I) Gini Coefficient: Ord4
1.00
1.00
0.80
Gini Coefficient
TCRE
0.60
0.40
7SSE
5SE
0.20
0.00
Trust Parameter
Trust Parameter y
Rounds after Intervention
(m) Decomp. AI: Ord5
(n) Decomp. CL: Ord5
(o) Gini Coefficient: Ord5
Figure 9: Repeats plots from Fig. 3 for 5 additional total orderings.
26

---

Counterfactual Effeet Decomposition in Multi-Agent Sequential Decision Making
We repeat experiments from Section 6.2 for randomly selected additional total orderings. The results from these experiments depicted in Fig. 9. From the plots corresponding to any of these total orderings,
draw similar conclusions to the ones drew from Fig. 3, especially from the plots that show the average percentage decomposition. We can conclude then that the empirical findings in Section 6.2 are robust to the uncertainty over the underlying total orderings.
N. Graphical Illustration of Counterfactual Effects from Sections 2 and 3
Treatment E
A:
Ho
Hi
S
%
(a) B[[,,,, |T]
Treatment
Ao
He
S
(b) B|Y [T; M Magdoor
Treatment
(c) E[Y,,, [T; M Mggoo))
Figure 10: Depicts all counterfactual estimates appearing in Definitions 2.1 (TCFE), 3.1 (tot-ASE), 3.2 (SSE) and Equation 2 (r-SSE) using the Sepsis example from the introduction section of (Triantafyllou et al., 2024). The decision decision-making making setting of this example is the same as the one from Section I and Section 6.2, but restricted to only two time-steps. We repeat the premise the example and necessary notation for completeness. Squares in the graphs denote agents' actions, for Al and H for clinician. Circles S are patient states, while S include both S and A, i.e., S (S, A). Y denotes the patient outcome after two time-steps. Edges that are striked through represent deactivated edges. Exogenous arrows represent interventions Ag that fix its value scenario, the former represents the action that was observed in the factual scenario (T), while the latter is the alternative treatment (0;,)) whose counterfactual effect, on Y, we analyze. A cyan colored node signifies that the node is set to the action that the agent took in the factual scenario, i.c., under treatment C. A magenta colored node signifies that the node set to the (counterfactual) action that the agent would have naturally taken under intervention E. Lastly, in Plot 10b I= {A := Ay, ((a.)) Iee (1.......)) (Definition 3.1), while in Plot 10c I = {Av,e == Ar,e(r(Ae hre (te.....))) (Definition 3.2).
of two actions, Treatment C or Treatment E. In the considered
27


Reviews:
[{"review_id": "wVQ2IudYB9", "reviewer": "Reviewer_rRtY", "summary": "This paper proposes a novel decomposition of counterfactual effects in multi-agent sequential decision-making settings. Building upon prior work on agent-specific counterfactual effects (cf-ASE), the authors present a bi-level decomposition separating the impact of an agent’s action into (i) how it affects the reachability of outcome-relevant states (r-SSE), and (ii) how much it contributes once those states are reached (ICC).\n\n## Update after rebuttal:\nI have read the authors' rebuttal. As I initially gave a weak acceptance, I have decided to maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We are glad to see that you find our proposed method clear, logically appropriate and enhancing the interpretability of decision-making outcomes. We are also happy to see that you find our experimental testbed well-suited. Please find below our response to your comments and questions.\n\n## Response to Comments and Questions\n\n**Scalability and Generalizability.** Since Reviewer LcKu had a similar comment, we would like to kindly point you to our response to their review.\n\n**Comparison to ASE.** Our approach to decomposing tot-ASE, termed ASE-SV, builds on the notion of agent-specific effects (ASE) introduced by (Triantafyllou et al., 2024). Fortunately, ASE is already well-defined in the MMDP setting, allowing us to leverage this concept directly. However, it is crucial to clarify that simply summing the agent-specific effects of individual agents does not yield tot-ASE. In fact, our experiments in the Sepsis environment reveal discrepancies of up to 95% in certain scenarios. To derive an *efficient* attribution method, we formulated the problem as a *cooperative game*. This formulation enabled us to define Shapley value in the setting of agent-specific effects. Furthermore, in Section 5 and Appendix F, we explain and formally define in this context a set of well-known desirable properties uniquely satisfied by Shapley value.\n\nCrucially, our work addresses a newly posed and complex problem that is clearly distinct, though related, from the focus of (Triantafyllou et al., 2024). Our key contribution lies in integrating concepts from a wide range of fields, including multi-agent systems (MMDPs), counterfactual reasoning (ASE), mediation analysis (Theorem 3.3), game theory (ASE-SV), information theory (r-SSE-ICC) and more, to tackle this novel challenge.\n\n**Notation Table.** Thank you for your suggestion. We would like to kindly point out that there already exists a notation table in Appendix B of our paper. \n\n**Q1.** While we have not explicitly tested how increasing the number of agents affects the practical performance of our approach, we speculate that scaling to larger multi-agent systems may require drawing more samples from the posterior distribution to ensure reliable counterfactual inference. This is because a larger number of agents increases the dimensionality of the joint action space, which in turn could introduce greater variability in our counterfactual estimates.\n\n**Q2.** Appendix J.2 of our paper demonstrates how the proposed decomposition can be applied to accountable decision making, specifically in the context of *blame attribution* in multi-agent systems. Revisiting the Sepsis scenario from Section 1, we illustrate how our method can be used to determine the proportion of blame attributable to the clinician for their action at time-step 10. Note that this level of fine-grained accountability assessment cannot be achieved by solely examining the total effect and ASE.\n\n**Q3.** First, we would like to clarify that in our Sepsis experiment, we do **not** assume access to the underlying causal model. Instead, our method relies solely on observational distributions. To enable counterfactual inference in this setting, we assume that noise monotonicity holds w.r.t. a chosen set of total orderings. To assess the robustness of our findings in the presence of potential violations of this assumption, we repeated the experiment from Section 6.2 across 5 randomly selected additional total orderings. The full results of this analysis are reported in Appendix N.\n\nThese results suggest that while the accuracy of individual counterfactual estimates can degrade when our causal assumptions are violated, the overall conclusions drawn from our analysis in Section 6.2 remain largely robust. This is encouraging, as it provides evidence supporting that our causal explanation formula (including r-SSE and ICC) can yield valuable insights in simulated domains where: (a) effect sizes and randomness reflect real-world settings, and (b) our theoretical assumptions might not hold.\n\n**Q4.** This is an intriguing question, and one we have actually also considered ourselves. The short answer is yes: our effect decomposition framework can be extended to handle dynamic control variables by integrating them into the SCM. One way to do this is by modeling the underlying MMDP as a *mechanised SCM* (Kenton et al., 2023). In this formulation, dynamic control variables can be represented as *mechanism variables* controlling the functional behavior of the MMDP variables. For example, in the Sepsis environment, the clinician's trust level could be modeled not only as evolving over time, but also as being influenced by changes in the AI agent's policy.\n\nKenton, Zachary, et al. \"Discovering agents.\" Artificial Intelligence. 2023.\n\n## Conclusion\n\nThank you again for your comments and questions. We would be happy to answer anything else in addition.", "claims_and_evidence": "The authors present a bi-level decomposition of counterfactual effects into reachability (r-SSE) and individual contribution (ICC) terms and formalize this using Structural Causal Models in MMDP settings.\nClear theoretical definitions and mathematical formulation have been shown, and empirical evaluation in environments previously used in causal multi-agent studies (e.g., Sepsis), shows that the decomposition captures interpretable trends across different trust parameters.\nHowever, one caveat is that some claims about practical applicability (e.g., potential use in real-world accountability systems) remain speculative since the experiments are limited to simulated domains, and assumptions such as noise monotonicity may not always hold in practice.", "methods_and_evaluation_criteria": "The method builds on Structural Causal Models over MMDPs, and introduces a bi-level decomposition into r-SSE (state reachability) and ICC (individual causal contribution). This method is conceptually grounded in prior causal inference frameworks and is logically appropriate for the multi-agent sequential setting.\n\nFor evaluation, the authors use two established benchmark environments — Graph and Sepsis — which are well-suited for modeling agent interdependencies and measuring counterfactual effects. Especially, the Sepsis environment (AI-clinician trust model) is directly relevant to applications in human-AI collaboration and accountability.\n\nThe limitation is that the experimental evaluation is focused only on simulated environments, and does not include other complex multi-agent RL benchmarks.", "theoretical_claims": "The introduced theoretical claims appear valid and aligned with established causal inference literature.", "experimental_designs_or_analyses": "The analysis includes variation across noise models, averaging over rollouts, and decomposition term breakdowns. The error bars and trend lines are presented clearly. In Figure 4, the authors demonstrate how ICC and reachability vary independently, which supports their claim that both aspects are necessary for full interpretation.\n\nHowever, the environments are simulated and relatively low-dimensional. While the designs are sound for demonstrating the proposed decomposition, it remains unclear how well the method would generalize to high-dimensional and/or real-world domains.", "relation_to_broader_scientific_literature": "This work extends Agent-Specific Effects (ASE) and counterfactual ASE (cf-ASE), introduced in the authors’ prior work (ICML 2024). The current paper decomposes the previously aggregated cf-ASE into two interpretable components,  r-SSE (state reachability) and ICC (individual causal contribution), enhancing the interpretability of causal influence.\n\nThis paper leverages SCM formalism applied to sequential settings, building on prior causal RL literature, which uses SCMs to define interventional queries in dynamic systems. However, previous works typically focus on single-agent or non-decomposed effects, whereas this work targets multi-agent interactions with decomposition.\n\nThe decomposition proposed in this paper contributes to the growing interest in interpretable multi-agent RL and causal accountability, allowing system designers to trace which agents are causally responsible for which parts of an outcome, a key issue in safety-critical or social contexts.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "+ Strengths\n\n> The paper builds upon prior work (cf-ASE) and introduces a bi-level decomposition that adds interpretability and granularity to multi-agent causal analysis. While incremental, the conceptual refinement is practically meaningful.\n\n+ Weaknesses\n\n> The evaluation is restricted to relatively simple domains (e.g., Sepsis, Graph), and the decomposition is not tested in complex multi-agent RL benchmarks or real-world tasks. This limits the generalizability of the conclusions.\n\n>  Although well-executed, the core idea is a decomposition of a quantity (cf-ASE) already introduced by the same authors. It seems to be regarded as an incremental extension.\n\n> The paper may be difficult to fully appreciate without reading their ICML 2024 paper, as many definitions and motivations are tightly linked to cf-ASE. This slightly affects self-containment.", "other_comments_or_suggestions": "Some key notations appear before being fully explained. Consider adding a notation table or briefly introducing them earlier for clarity.", "questions_for_authors": "While the appendix discusses the computational complexity with the increasing number of agents N, are there other challenges (beyond computation) that may arise when scaling the decomposition to large multi-agent systems?\n\nYour proposed decomposition allows for quantitatively separating agent or state influence into r-SSE and ICC. Could the authors provide evidence or examples showing that this separation leads to different decisions or policy adjustments, compared to using cf-ASE alone?\n\nThe method assumes access to an accurate SCM or simulator, but real-world models may be misspecified. How robust are r-SSE and ICC to such inaccuracies in structural functions?\n\nWhile it is common to treat experimental control variables (like the trust level in Sepsis) as fixed scalars, some recent work considers them as a dynamic or latent quantity that evolves over time. Do the authors see the potential for extending their decomposition framework to handle such dynamic control variables, possibly by integrating them into the SCM or policy structure?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBZajcnE3P", "reviewer": "Reviewer_Lft9", "summary": "This paper focuses on the causal analysis of decision-making in multi-agent cooperative frameworks, specifically the decomposition and attribution of counterfactual effects in the decision-making process of agents. The key contribution is the decomposition of the total counterfactual effect into two parts: the agent-specific effect, which propagates through the subsequent behaviors of other agents, and the state-specific effect, which propagates through environmental state transitions. Furthermore, the authors refine the attribution of the agent-specific effect by leveraging the Shapley value to distribute it among individual agents. The proposed causal analysis method is validated in two experimental environments, demonstrating the feasibility of the counterfactual effect decomposition approach.\n\n### update after rebuttal:\nI thank the authors for their response and have taken into account the perspectives of the other reviewers.  I will maintain my score, as it is already a positive one.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive score and your kind words about our work. We are glad to hear that you find our approach well-supported, and appreciate its novelty, intuitiveness, and the comprehensive causal analysis it provides for multi-agent decision-making. We are also happy to see that you find our experimental evaluation reasonable and providing sufficient evidence to validate our approach.\n\nThank you again for all your positive comments. We are happy to answer any further questions or comments you might have.", "claims_and_evidence": "The paper's main claim is that its proposed causal analysis method can decompose the counterfactual effect in multi-agent decision processes into effects propagated through agent behaviors and effects propagated through state transitions. The main claim is well-supported by both clear theoretical justifications and empirical results from two experiments, providing sufficient evidence to validate the proposed approach.", "methods_and_evaluation_criteria": "I think the proposed methods and evaluation criteria are appropriate. The approach is intuitive, systematically decomposing the total counterfactual effect into agent-specific and state-specific effects, followed by further attribution using Shapley values and ICC. This provides a comprehensive causal analysis framework for multi-agent decision-making.", "theoretical_claims": "I reviewed the theoretical proofs regarding the decomposition of counterfactual effects, and overall, the proofs appear complete and sound.", "experimental_designs_or_analyses": "I find the experimental setups in the Gridworld and Sepsis environments to be reasonable. Both environments are designed with well-defined agent interactions, and the authors employed posterior sampling and attribution methods to validate the robustness of their proposed framework.", "relation_to_broader_scientific_literature": "One of the paper’s main contributions is its novel causal analysis perspective on multi-agent cooperation frameworks. Most existing multi-agent cooperation research focuses on designing coordination mechanisms to improve performance on specific tasks. In contrast, this paper provides an analytical tool for attributing decision-making outcomes, which could further enhance the design of cooperative frameworks by enabling a more detailed understanding of agent interactions.", "essential_references_not_discussed": "The paper discusses a sufficient number of related works.", "other_strengths_and_weaknesses": "One notable innovation in this paper is the introduction of Shapley values to further decompose the total agent-specific effect at the individual agent level. This approach adds granularity to counterfactual effect attribution.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTwHc2Dej7", "reviewer": "Reviewer_LcKu", "summary": "The paper proposes a causal explanation formula for multi-agent Markov Decision Processes (MMDPs). It uses Structural Causal Models (SCMs) to decomposes the total counterfactual effect of an agent's action by attributing to each agent and state variable a score that results from their respective contributions to the outcome. Shapley values have been used  to attribute the total effect to individual agents.\nSimilarly, intrinsic causal contributions (ICC) have been used to decompose the reverse state-specific effect (r-SSE). Experiments were conducted on Gridworld environment with LLM-assisted agents and a sepsis management simulator.\n\n## update after rebuttal: \nI read the rebuttal of the authors as well as the questions of the reviewers. I maintain my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and positive score. We are glad to see that you find our paper well-written and addressing significant issues of AI systems. We are also happy to hear that you find our method insightful and sensible, and our experimental setup sound and clear. Please find below our response to your comments and questions.\n\n## Response to Comments and Questions\n\n**Scalability and Generalizability.** We recognize that improving and evaluating the scalability of our approach would be important for its practical applicability in systems with larger numbers of agents. However, we view this as an independent research challenge that, while important, lies somewhat outside the main scope of this paper. Our primary focus here is to provide an interpretable solution to the problem of counterfactual effect decomposition in multi-agent MDPs. \n\nThat said, we note that there are multiple impactful multi-agent settings that naturally involve only a small number of agents, e.g., AI assistants or AI supervision. Nonetheless, empirically evaluating the strategies we propose in Appendix I for mitigating computational complexity, within our setting, represents a promising direction for future work.\n\n**Identifiability Assumptions.** The noise monotonicity assumption imposes a structural restriction on the counterfactual distribution of the causal model. Similar to other causal assumptions, such as *monotonicity* in binary models (Pearl, 1999), noise monotonicity cannot be verified from observational or interventional data. Therefore, assessing its validity in real-world (non-simulated) settings remains challenging.\n\nAs you note in your review, we include an empirical robustness analysis in our paper to assess the sensitivity of our findings in the Sepsis experiment to potential violations of noise monotonicity. The results of this analysis are reported in Appendix N.\n\nIn Appendix J.1, we discuss extending the applicability of our effect decomposition approach to non-identifiable domains through *partial counterfactual identification* (Manksi, 1990). Rather than relying on point estimates, this approach would operate with bounds on counterfactual quantities. As such, the practical applicability of our method will broaden, albeit on the expense of *efficiency*, i.e., not attributing the full effect.\n\nA partial identification method directly compatible with our setting was recently proposed by (Zhang et al., 2022). Although assessing and enhancing the informativeness of their derived bounds, as well as examining the scalability of their approach in our context, lie beyond the scope of this paper, we regard this as a promising and important avenue for future research.\n\n## Conclusion\n\nWe thank you again for your comments and questions. We would be happy to answer anything else in addition.\n\n## References\n\nPearl, Judea. \"Probabilities of causation: Three counterfactual interpretations and their identification.\" Synthese. 1999.\n\nManski, Charles F. \"Nonparametric bounds on treatment effects.\" The American Economic Review. 1990.\n\nZhang, Junzhe, Jin Tian, and Elias Bareinboim. \"Partial counterfactual identification from observational and experimental data.\" ICML. 2022.", "claims_and_evidence": "The main claim of the paper is that counterfactual effects of an agent's action in multi-agent sequential decision-making can be decomposed into agent-specific and state-specific contributions. This provides insights into accountability.\nThe claim is theoretically justified by theorems and proofs in the Appendix. Experiments support the claims; evaluations of the estimation error of the results and the robustness of the noise monotonicity are provided in the Appendix", "methods_and_evaluation_criteria": "The proposed methods makes sense for the problem of counterfactual effect decomposition in multi-agent systems.\nThe methods, Shapley values and ICC, provide fair attribution and accountability, and are theoretically sound. \n Evaluation on benchmark datasets (Gridworld and Sepsis) are suitable to the problem.", "theoretical_claims": "I checked the proofs of theorem 3.3. and theorem 4.2. in Appendix H. They look sound to me.", "experimental_designs_or_analyses": "The experimental designs are sound and clear. The validity of the counterfactual effects have been done on multiple trials. Computational complexity has been discussed. I checked both the Gridworld and the Sepsis experiments.", "relation_to_broader_scientific_literature": "The paper draws ideas from various fields, including mediation analysis, intrinsic causal contributions  (Janzing et al), Shapley values, blame attribution (Halpern et al).", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "The paper addresses significant issues: accountability and explainability of AI systems. It is well written.\nOne potential weakness would be scalability and the computational complexity involved. To that end, experiments on real-world use cases with many agents would be helpful.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The assumption of noise monotonicity guarantees counterfactual identifiability. How realistic is this assumption in real-world use cases and how would the paper's framework treat the cases where the assumption is violated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RyFGZFO7C8", "reviewer": "Reviewer_n1oL", "summary": "This paper studies the attribution of counterfactual outcome to agent actions and states, that are total agent-specific effect and reverse state-specific effect. Moreover, it futher decompose the total agent-specific effect into individual agent effect, and the reverse state-specific\neffect (r-SSE) into r-SSE-ICC. Through experiments on the Gridworld environment and sepsis management simulator, the interpretability of the method is demonstrated.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Please find below our response to your comments and questions.\n\n## Response to Comments and Questions\n\n**Proof of Theorem 3.3.** We respectfully disagree with your concern regarding the correctness of the proof of Theorem 3.3. While we understand that the result may initially seem surprising, the proof follows in a straightforward manner. The first step is licensed by Definition 2.1, while the third step by Definition 3.1 and Equation (2). In the second step, we apply a standard add-and-subtract trick by introducing $\\mathrm{E}[Y|\\tau;M]_{M^{do(I)}}$ into the expression.\n\n**Interpretability.** Causal mediation analysis consists a well-established approach to interpreting the effect of an exposure variable to a response variable, by analyzing how it propagates through causal paths. Based on a similar principle, our proposed approach explains the effect of an agent's action on the outcome of an MMDP, by analyzing how it propagates through its influence on the agents’ behavior and the environment dynamics. Additionally, we have evaluated the interpretability of our decomposition approach by conducting extensive experiments on two RL environments featuring heterogeneous agents and diverse interaction protocols.\nThe results of these evaluations indicate that our approach yields interpretable insights and aligns well with standard intuitions.\n\nCould we kindly ask the reviewer to elaborate on why they think our approach and evaluation protocol are not suitable for interpretability? If there are any specific concerns regarding our method, we would be glad to hear them and if possible try to also address them.\n\n**Equation (2).** Let $\\tau$ be a trajectory generated by the MMDP-SCM $M$. Both $\\mathrm{E}[Y_{a_{i,t}}|\\tau]\\_{M}$ and $\\mathrm{E}[Y|\\tau;M]\\_{M^{do(I)}}$ measure the expected counterfactual value of outcome $Y$ in $\\tau$, but under different interventions. The former corresponds to the intervention $do(A_{i,t} := a_{i,t})$, which fixes the action of agent $i$ at time-step $t$ to $a_{i,t}$. The latter corresponds to the intervention $do(I)$, which fixes all agents' actions, following time-step $t$, to the values that they would naturally take under intervention $do(A_{i,t} := a_{i,t})$.\n\nIf it helps, we would like to kindly point out that in Section 6.1 of our paper, and particularly in the paragraph **Counterfactual effects**, you can find a detailed explanation of all introduced counterfactual measures, including r-SSE (Eq. 2), in the context of our Gridworld experiment. Furthermore, Appendix O provides a graphical illustration of all counterfactual estimates appearing in our paper using the Sepsis example from the introduction section of (Triantafyllou et al., 2024) and the causal graph used therein.\n\n**Definition 5.2.** $\\text{ASE}^{\\\\{1, ..., n\\\\}}$ corresponds to the total agent-specific effect, which quantifies the effect of an intervention that propagates through all the agents. In contrast, $\\text{ASE}^{S \\cup \\\\{j\\\\}}$ quantifies the effect that propagates through only the agents included in the subset $S \\cup \\\\{j\\\\} \\subseteq \\\\{1, ..., n\\\\}$. Note that this distinction is explicitly highlighted in lines 283-290 (left column) of our manuscript.\n\n## Conclusion\n\nWe thank you again for your comments and questions. We would be happy to answer anything else in addition.", "claims_and_evidence": "This paper try to explain the counterfactual effect of agent action. To achieve this, this paper defines a series of metrics such as tot-ASE. However, the definitions are complicated and not easy to understand. I observe the experimental results is mainly composed of measuring these metrics. I think the work does not accomplish the goal of interpretability.", "methods_and_evaluation_criteria": "The same to the previous part. The proposed method and evaluation criteria do not improve the interpretability of the agent actions.", "theoretical_claims": "I chech the proof of Theorem 3.3, which I think is the cornerstone of the paper. However, I think the proof is wrong. The third equation does not relate the Definition 3.1 and Definition 3.2.", "experimental_designs_or_analyses": "I chech the experimental parts. The main concern is still that the results are composed of the proposed metrics which is not interpretable enough. I suggest authors to relate the interpretability result to some quantitative analysis which is easy to understand by human.", "relation_to_broader_scientific_literature": "I think the contribution of the paper lies in the integration of causal media analysis and the multi-agent decision. Although it propose finer grained metrics of effect beyond Triantafyllou et al., 2024, I feel it does not help to better interpretability.", "essential_references_not_discussed": "I think the core references have been discussed and cited in the paper.", "other_strengths_and_weaknesses": "The main strengths and weaknesses have been discussed in the previous parts.", "other_comments_or_suggestions": "In the last equation of Eq. (2), the difference of (1) \\\\(\\\\mathbb{E}[Y|\\\\tau; M]_{M^{do(I)}}\\\\) and the other term is not clear.", "questions_for_authors": "In Definition 5.2, the definition of $ASE^{S\\cup \\\\{j\\\\}}$ is not clear, because the previous definition of ASE is designed for $\\\\{1,2,3,...,n\\\\}$", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a causal explanation framework that decomposes the total counterfactual effect of an agent’s action in multi-agent MDPs. Among the four reviewers, three provided positive evaluations: 2 Accept and 1 Weak Accept. One reviewer raised concerns regarding Theorem, which were addressed step-by-step during the rebuttal; another reviewer explicitly commented that the proof did not appear incorrect. 
The authors responded to reviewer concerns throughout the rebuttal phase, though some issues such as the limited experimental scope and the scalability of the approach in real-world settings remain only partially addressed. Taking these factors into account, I would recommend Weak Accept.
Decision: Accept (poster)