ICML 2025

Counterfactual Effect Decomposition in
Multi-Agent Sequential Decision Making
Stelios Triantafyllou 1 Aleksa Sukovic 1 Yasaman Zolfimoselo 1 Goran Radanovic 1
Abstract
We address the challenge of explaining counter-
factual outcomes in multi-agent Markov decision
processes. In particular, we aim to explain the to-
tal counterfactual effect of an agent’s action on the
outcome of a realized scenario through its influ-
ence on the environment dynamics and the agents’
behavior. To achieve this, we introduce a novel
causal explanation formula that decomposes the
counterfactual effect by attributing to each agent
and state variable a score reflecting their respec-
tive contributions to the effect. First, we show that
the total counterfactual effect of an agent’s action
can be decomposed into two components: one
measuring the effect that propagates through all
subsequent agents’ actions and another related
to the effect that propagates through the state
transitions. Building on recent advancements in
causal contribution analysis, we further decom-
pose these two effects as follows. For the former,
we consider agent-specific effects – a causal con-
cept that quantifies the counterfactual effect of an
agent’s action that propagates through a subset
of agents. Based on this notion, we use Shapley
value to attribute the effect to individual agents.
For the latter, we consider the concept of structure-
preserving interventions and attribute the effect
to state variables based on their “intrinsic” contri-
butions. Through extensive experimentation, we
demonstrate the interpretability of our approach
in a Gridworld environment with LLM-assisted
agents and a sepsis management simulator.
1. Introduction
Applying counterfactual reasoning to retrospectively an-
alyze the impact of different actions in decision making
1Max Planck Institute for Software Systems, Germany. Corre-
spondence to: Stelios Triantafyllou <strianta@mpi-sws.org>.
Proceedings of the 42 nd International Conference...