When Should Reinforcement Learning Use Causal Reasoning?
Authors: Oliver Schulte
Pascal Poupart
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL) and causal reasoning naturally complement each other. The goal of causal reasoning is to predict the effects of interventions in an environment, while the goal of reinforcement learning is to select interventions that maximize the rewards the agent receives from the environment. Reinforcement learning includes the two most powerful sources of information for estimating causal relationships: temporal ordering and the ability to act on an environment. This paper provides a theoretical study examining which reinforcement learning settings  we can expect to benefit from causal reasoning, and how. According to our analysis, the key factor is {\em whether the behavioral policy---which generates the data---can be executed by the learning agent}, meaning that the observation signal available to the learning agent comprises all observations used by the behavioral policy. Common RL settings with behavioral policies that are executable by the learning agent include on-policy learning 
 and online exploration, where the learning agent uses a behavioral policy to explore the environment. 
 Common RL settings with behavioral policies that are not executable by the learning agent include offline learning with a partially observable state space and asymmetric imitation learning where the demonstrator has access to more observations than the imitator. Using the theory of causal graphs, we show formally that when the behavioral policy is executable by the learning agent, conditional probabilities are causal, and can therefore be used to estimate expected rewards as done in traditional RL. However, when the behavioral policy is not executable by the learning agent, conditional probabilities may be confounded and provide misleading estimates of expected rewards. For confounded settings, we describe previous and new methods for leveraging causal reasoning.

Published in Transactions on Machine Learning Research (April/2025)
When Should Reinforcement Learning Use Causal Reason- ing?
Oliver Schulte
oschulte@cs.sfu.cc
School of Computing Science, Simon Praser University, Voncouver, Canada
Pascal Poupart
ypoupart@vwaterloo.co
Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada Vector Institute, Toronta, Canada
Reviewed on OpenReview: ht tp s: 0/ en en ew in et /f or um 74 d- DI uk 8Z
Abstract
Reinforcement learning (RL) and causal reasoning naturally complement each other. The goal of causal reasoning is to predict the effects of interventions in an environment, while the goal of reinforcement learning is to select interventions that maximize the rewards the agent receives from the environment. Reinforcement learning includes the two most pow- erful sources information for estimating causal relationships: temporal ordering and the ability to on an environment. This paper provides a theoretical study examining which reinforcement learning settings According to our analysis, the key factor is whether the behavioral poliey-which generates the data-cas can be erecuted by the learning agent, meaning that the observation signal avail- able to the learning agent comprises all observations used by the behavioral policy. Common RL settings with behavioral policies that are executable by the learning agent include on- policy learning and online exploration, where the learning agent uses a behavioral policy to explore the environment. Common RL scttings with behavioral policies that are not exo- cutable by the learning agent include ofllne learning with a partially observable state space and asymmetric imitation learning where the demonstrator has access to more observations than the imitator. Using the theory of causal graphs, we show formally that when the behavioral policy is executable by the learning agent, conditional probabilities are causal, and can therefore be used to estimate expected rewards as done in traditional RT. However, when the behavioral policy is not executable by the learning agent, conditional probabilities may be confounded and provide mislcading estimates of expected rewards. For confounded settings, we describe previous and new methods for leveraging causal reasoning.
can expect benefit from causal reasoning, and how.
1 Introduction: Causal Probabilities in Reinforcement Learning
The goal of decision-making in Markov Decision Process (MDP) intervene the environment maximize the agent's cunulative reward. A key insight of causal decision theory is that the impact of action should be estimated as causal effect, not correlation. Visits to the doctor correlate with illnesses, but avoiding seeing doctor does not make patient healthier (Pearl, 2000, Ch.4.1.1). Several causality rescarchers have therefore argued that reinforcement lcarning can benefit from adopting causal models to predict the elfect of actions. This article is directed towards reinforcement learning researchers who want to explore the use causal models. We provide conceptual and theoretical foundations to facilitate the adoption of causal modcls by reinforcement Icarning rescarchers. We use as much as possible terminology, notation, and examples from reinforcement learning. A running example gives explicit computations that illustrate causal concepts. This paper therefore serve as short. tutorial on causal modeling for RT.

---

Published in Transactions on Machine Learning Research (April/2025)
Table 1: A four-level causal hierarchy, which refines Pearl's three-level hierarchy Associatior-nntervnttion Counterfactual (Pearl, 2000). Our analysis shows that with executable behavorial policies, queries of the first three types con be computed from conditional probabiliites.
Level
Notation P(RSS, A)
Typical Question
Example
Association
What reward fol- lows after an agent chooses A?
How often does a shot lead to goal?
Intervention
P(RS,, do(A))
If choose A, what. will my reward be? What if I had cho- sen A instead of B?
If take a shot, will I score goal?
What-if
P(RA(S,B)
What if had taken a shot instead of making a pass?
Counterfactual
Hindsight
P(RA|S,B, B,
How would my re- ward change had chosen A in- stead of B?
failed to score. What if had taken a shot instead of making pass?
Counterfactual
roscarchers. An excellent long tutorial is provided by Barcinboim (2020), recent survey by Deng et al. (2023), and book-length treatment by Bareinboim et al. (2025).
Conditional probabilities measure the strength of associations or correlations, but not necessarily the causal effect of an action. Using Pearl's do operator, the causal effect of setting variable A to the value a given evidence covariates X can be written as a conditional probability of the form P(Y]do(A- = a), X = x)). (The formal semantics for the do operator is defined in Section 2 below.) Since causal effects are based on interventions, we refer them also as interventional probabilities. In the medical visit example, the strong correlation means that P(Iliness) Visit) is high. However, making person visit the doctor has causal effect on their illness, so we have P(Nlines] Visit) >> P(Iliness]do) do( Visit) P P(Iliness). The question we address is under what conditions causal probabiliiies provide a different approach to reinforcement learning than conditional probabilities.
1.1 Summary of Main Result
Our analysis follows a "ladder of causation" as described by (Pearl, 2000): A hierarchy of probabilistic statements that require causal reasoning of increasing complexity. The levels correspond to associations, interventions, and counterfactuals. Table 1 illustrates these concepts in the RL setting. A formal semantics for each type of probability can be defined terms a generative model that is based on a causal graph. Among counterfactuals, we distinguish between what-if queries and hindsight queries. A what-if query concerns the results of deviating from an action taken; an example from sports domain would be "What if I had taken shot instead of making a pass?". A hindsight query conditions on observed outcome. An example of a hindsight query would be "T failed to score. What if I had taken a shot instead of making pass?". Each level of the causal hierarchy supports a different policy evaluation method based on different types of reward and transition probabililies. For example, conditional reward is given by the conditional probability P( P(Re+1lA, = a, St) while the interventional reward given by the interventional probability P(R++1 |do A, a,), St), where P(RetalA, a,, St) the conditional probability of receiving reward R time & + 1 given action A, and state S, at time t.
We define four Bellman equations for Q-functions based (i) standard conditional probabilities, (ii) in- terventional probabilities, (iii) what-if probabilities, and (iv) hindsight probabilities. Our main theoretical result states that if the behavioral data-generating policy is executable by the learning agent, then conditional Q-values, interventional Q-values, and whal-if Q-values are equivalent. Thus under executability, causal Q-values can be correctly estimated from conditional probabilities, as is done in standard RL.
By "executable" we mean that the observation signal available the learning agent comprises all observations used by the behavioral policy. Figure illustrates the concept. in a simple sports setting (such as hockey
2

---

Published in Transactions on Machine Learning Research (April/2025)
player health
close goal
goalle health
player health
close to goal
goalie health
shoot
Spgres gool
shoot
Spgrrss goal-
(a) Ouline View.
(b) Offline View.
Figure Causal Graphs for a sports scenario like hockey or soccer to illustrate executability. We follow the conventions of influence diagrams distinguish state variables, actions, and rewards. All variables are binary. Variables observed by the learning agent are gray, latent variables white. Whether a player takes shot depends on their location and whether they are injured. Likewise, the chance of their shot leading to goal depends on their location and health. Thus Player Health is common cause of the action and reward. Figure la: In the online setting, the athlete learns from their own experience, which includes their health. Therefore a behavioral policy that depends on the agent's 's health is executable. Figure 1b: In the offline setting, the learner is different from the athlete, for example a coach, and docs not observe the health of the behavioral agent. Therefore a behavioral policy that depends on the agent's 's health not executable. Player health is an unobserved confounder of action and reward.
what-if interventional conditional Q-values
RT. setting
Executable?
Reason
On-policy
Learned policy behavioral policy
Online exploration
Tearner executes behavioral policy
Learner's observation space
Complete Observability
behavioral agent's observation space
Off-policy and
Behavioral policy relies on observations that are not observable to the learner
Offline learning and partial observability
X
Table 2: Reinforcement learning settings in which we can expect conditional probabilities to be equivalent to interventional probabilitics and to what-if counterfactual probabilitics. Online exploration refers to a setting in which the learning agent execules a behavioral policy to explore the enviromment. Hindsight counterfactuals cannot be reduced conditional probabilities in any setting.
or soccer). Player health common cause of the player's decision (c.g., to shoot) and the player's success (score a goal). In the first-person online setting, the athlete is aware of their own health, and execute policy that depends on the athlete's health (e.g., "shoot only if healthy"). In the third-person offline setting, observer such coach, does not have access to the athlete's health. They can execute only policies that depend on whether a player is close to the goal, but not those that depend on whether the player is healthy.
We show by example that hindsight Q-values differ from conditional Q-values even under executability. The fundamental reason is that future information provides extra information about current latent states over and above the past history. The special power of hindsight has been noted in previous RL work, such as the well-known hindsight experience replay approach (Andrychowicz et al., 2017), hindsight credit assignment (Harulyunyan et al., 2019), and data augmentation using hindsight (Sun et al., 2024).
1.2 Implications for RL settings
Next we discuss which RI. settings satisfy the executability condition. Our overall conclusion, illustrated Table 2, is that what-if and interventional probabilities differ from conditional probabilities only in the offline off-policy RL selling with partial obsercability (the POOO setling).
RL settings with exccutable behavioral policies. Common RL settings with behavioral policies that executable by the learning agent include on-policy learning and online exploration, where the learning
3

---

Published in Transactions on Machine Learning Research (April/2025)
agent uses behavioral policy to explore the environment (e.g., e-greedy learning); see Appendix Figure 6 for illustration. In on-policy learning the behavioral and the learned policy coincide so they clearly share the same observation signal. Accordingly, Schölkopf et al. (2021) note that "[Reinforcement learning] sometimes effectively directly estimates do-probabilities. E.g., on-policy learning estimates do-probabílitics for the interventions specified by the policy". Our work formalizes their observation that the on-policy setting is sufficient for conditional and interventional probabilities to coincide, and goes beyond it to consider other RL settings, such as the following.
In online exploration, the learning agent directly interacts with its enviromment to generate data and learn from their own experience exploring the environment (see Figure 6). For example a video game playing system can execute actions in the game and observe their effects (Mnih et al., 2015). Since the behavioral policy is executed by the learning agent to explore the environment, it must be executable. In the video game example, the game playing agent employs an exploration policy, the policy must depend only on the information available in the video game.
Another sufficient condition for executability is complete obsereability, where the environment is completely observable for both the behavioral and the learning agent. For instance, the first phase of training the AlphaGo system was based on an offiine dataset of games by Go masters (Silver et al., 2016). Go is a completely observable board game with no hidden information. Under complete observability, the learning agent has access to the same observations as the behavioral agent, as required for executability.
The Go example also illustrates how imitation learning can satisfy the executability condition if both the demonstrator and the learning agent (trainee), who aims to imitate them, share the same observation space. For illustration in online imitation learning, in the DAgger approach (Ross et al., 2011), the traince explores the state space using& mixture of the demonstrator's 's policy and the current learned policy. Using a mixture implies that both the expert and the trainee's policies operate with the same input observations.
RL settings without executable behavioral policies. When the behavioral policy is not executable by the learning agent, the learning agent may receive a different observation signal than the behavioral agent, as shown in Figure 2b. As we show in worked-out examples below, without executabilily conditional probabilities may be confounded by the learning agent's missing information, and provide misleading estimates of expected rewards. In the example of Figure 1b, Player Hcalth is not observable by the learning agent, and thus confounds the decisions of the behavioral agent (i.e., whether the athlete shoots) with the rewards obtained by the behavioral agent (i.e., whether the athlete scores).
In the offline setting, the learning agent cannot directly interact with its environment, and instead relics on an oflline datasel, which may not record all the observations available to the behavioral agent; see Figure 2. Several algorithms have been developed for imitation learning in a non-executable setting where the expert has access to more observations than the trainee. Ha et al. (2024) refer to this type of setting as privileged learning, whereas Warrington et al. (2021) use the term asymmetric imitation learning.
All results described so far hold even in the infinite sample setting with perfect probability estimates. With limited samples, confounding can become an issue even when the behavioral agent and the learning agent are one and the same. This is due to the implicit partial observability phenomenon described by Ghosh et al. (2021): when generalizing a policy from training contexts to a new test-time context, the agent may have less information at test time than at training time.' ! An example given by Ghosh et al. is navigating to the location of a larget object. using a map. The training data may allow the agent to train an image classifier with near-perfect accuracy to extract the location, whereas when tested in new context, the image classifier may provide insufficiently accurate information. During training time, the output of the image classifier is a common cause of the agent's actions (route chosen) and the reward (target found). Ghosh et al. suggest that because of limited samples, at test time the agent has only partial observability of the image information, which in contrast is perfectly observable during training. In our terminology, the test time agent is in a POO0 setting, and policy estimates based on conditional probabilities from the training data will be confounded.
We owe this point an anonymous reviewer.
4

---

Published in Transactions on Machine Learning Research (April/2025)
BAY
0
dkpoymen
coladid
(a) Offline RL: executable behavioral policies.
(b) Offline RL: non-executable behavioral policies.
Figure 2: Offline RL-learning employs a dataset D collected by some (potentially unknown) behavior policy Tg. The dataset collected once, and is not altered during training. The training process does not interact with the environment directly, and the policy is only deployed after being fully trained. Figure 2a (Levine et al., 2020): Offline RL with executability where the behavioral policy मh and the learned policy и are based the same observation signal. Figure 2b: Offline RL without executability where the behavior policy π has access to more observations than the learned policy
Section 5
RL settings in which the behavioral policy is is not executable by the learning agent
Section 4
If the behavioral policy executable by the learning agent, then Policy Evaluation with Conditional Probabiities Policy Evaluation with Interventional Probabiities Policy Evaluation with What-if Probabilities
Policy Evaluation
Policy Evaluation with Interventional Probabilities
Policy Evaluation with What-if Probabilitiss
with Conditional Probabilities
Section
Dynamic Decision Networks for POMDPs Causal Model for Representing Belief MDP
Section 2
Background: Structural Causal Models Definition of Causal Probabilities
Figure 3: Topies by paper section and connections between them. POMDP Partially Observable Markov Decision Process.
Paper Overview Figure illustrates our paper organization. Section gives background on causal models and how they define precise semantics for interventional and counterfactual probabilitics. Since latent confounders are the key difference between causation and correlation, partially observable Markov Decision Processes (POMDPs) are the relevant RL model for understanding the importance of causal reasoning. To port causal concepts to RL, we therefore require type of causal model that can represent POMDPs. Section 3 describes new variant of dynamic decision network (DDN) (Russell & Norvig, 2010, Ch.17) for representing POMDPs. In a DDN, an environment state is represented by an assignment. of values to state variables. DDN models causes both the agent's actions and the environment's responses.
Section shows how DDN supports model-based policy evaluation in three different ways: based on conditional probabilities, interventional probabilities, or what-if probabilities. Our main formal proposition
5

---

Published in Transactions on Machine Learning Research (April/2025)
Notation
Mcaning
Generic Random Variables. Nodes in a Causal Graph. 0uu 2
0
Observed Variables.
Z
Latent Variables
U
Source Variables in a causal graph.
X
X
Non-source variables with positive indegree
G
Directed Acyelic Graph (DAG); causal graph
Pax resp. Pai
Parents/ direct causes of variable resp. X;
C
Probabilistic Structural Causal Model
D
Dynamic Decision Network
fx resp. fi
Deterministic local function for variable resp. X;
5
State Space in POMDP
S
State Variables in factored POMDP: S 50 (A, R}
A
Intervention Target; Decision Variable in Influence Diagram; Action in POMDP Selecting action à as an intervention
do(A =
R
Reward
6(Z)
Belief State
a
Policy
(o, 6))
Probability of action given current observation and belief state
Q (0, 6),
Expected Return given current observation, belief state, and action
Q (0, 6), do()))
Expected Return given current observation, belief state, and aetion/intervention
Table 3: Notation used in this paper
states that when the behavioral policy is executable by the learning agent, policy evaluation with conditional probabilitics is equivalent to policy evaluation with Iatorventiona//hatte probabilitics and what-if counter- lactuals. Our results do not ussume that the learning agent given Irue causal model of the behavioral policy and the environmental process. As long as the learning agent's observation signal includes the causes of the behavioral agent's actions, conditional probabilities inferred from the action data are equivalent to interventional and what-if probabilities, matter what the true causal model is.
Section 5 shows how the causal policy evaluation theorem can be used to understand which RL settings do and do not satisfy executability, and therefore do or do not require causal policy evaluation. Our final section utilizes the concepts and insights developed in the main body to review related work on causal models in RL., and describe directions for future research from the perspective of confounding and executability.
2 Background: Structural Causal Models and Counterfactuals
Notation Table 3 sunmarizes the notation used in this paper, and previews the concepts introduced in the remainder.
Our goal this section is to define the class of structural cansal models, which give a precise semantics for interventional and counterfactual probabilities, collectively called causal probabilities.
2.1 Structural Causal Models
Structural causal models (SCMs) combine causal graph with latent variables and deterministic local func- tions that map parent values to child values (Pearl, 2000). SCMs are compatible with deep learning, in that the local functions are decoders in the sense of deep generative models. Deep generative models with latent variables are therefore a powerful architecture for implementing and learning models for counterfact rea- soning (Geffner et al., 2022). Our presentation SCMs follows those of Pearl (2000) and of Schölkopf et al. (2021).
A structural causal model (SCM) is pair S (G, F) meeting the following conditions.

---

Published in Transactions on Machine Learning Research (April/2025)
G' is DAG over random variables X. Let U be the set of source nodes with indegree 0 in G. We write X X U for the set of non-source variables.
F (S1 (. is set of local functions such that each J deterministically maps the parents of non-source variable X; E X to a value of X₁- The local functions are often written in the form of structural equation:
X, = f((pa.).
(1)
The souree variables are also called background variables or erogenous variables. They can be, and often are, latent variables. For example, in linear structural equation
f(X,e) = aX +b+
the parent X of Y represents an observed cause and the noise term can be modeled as latent parent summarizing unobserved causal inffuences. Given an assignment U = u of values to the source variables, we can compute values for the non-source variables X by using the local functions to assign values to the children of the source nodes, then to the children the children, etc. This recursive evaluation procedure defines a solution function
S (u)
(2)
where i is an assignment of values to the non-source nodes X. A probabilistic SCM is a pair C = (S,b) where b is a joint prior distribution over the source variables U (Pearl, 2000, 7.27. A probabilistic SCM defines a joint distribution over the variables X --
pC X E. U.=-Ul
P X 2 u) b(U u)
(3)
b( u) = I b(U =
(4)
ÚEU
1, if ps (u)) = E
pCp X
x U
u)
(5)
0, otherwise
where (u) the solution function (Equation (2).) Equation (4) says that the unconditional prior distri- bution over source variables factors into individual priors over each source variable. Equation (5) applies the deterministic solution function as deterministic decoder that maps the source variables to unique assignment for non-source variables. Section 3.2 below gives an example of an SCM.
As usual with generative models, a probabilistic SCM defines mixture distribution over observed variables through marginalization
Ps(
Σ
P(z)
(6)
2:
2.2 Interventional Probabilities
An interventional distribution (Pearl, 2000, Dfn. 7.1.3) is denoted P(Y|do( = a)), where Y and A comprise observed variables. It represents the cousal effect (Pearl, 2000, Dfn.3.2.1.) of intervening in the environment-the effect on variables Y of sctting a variable A to possible value à. In graphical terms, the effect of an intervention is computed by removing links pointing into A, fixing the values of X variables to the assignment à, then computing the joint probability of Y in the truncated model. Removing the parents of variable represents that an intervention removes the causal influence of the parents. For example, if the shooting decision of player is caused by their health and their distance to the goal, then if intervene make the player shoot, their decision is determined and no longer depends other variables.

---

Published in Transactions on Machine Learning Research (April/2025)
Remark notation. The à notation does not indicate quantity estimated from data, but intervention. We sometimes use the syntactic sugar to highlight a context where A is intervened upon. In our applications to RL, we consider intervening only on a special variable A that represents the agent's actions. However, the intervention semantics is well-defined for manipulating any random variable in a causal modcl, not only a designated special action/decision variable.
Formally, an intervention that assigns value variable A is represented by the truncated submodel S, AG Fa), which is the causal model where GA contains all edges in G except those pointing into variable A, and Fa (fi X, # A} u {A â}. Here {fr : X4 * A] is the set of all local functions for unmanipulated variables, and A = à is the constant function that assigns variable A its manipulated value. Similarly let ba be the prior distribution over source node variables that assigns probability 1 to and agrees with b on all other variables. Formally, ba(A- = @) = 1, and ba(U = u) = b(U = u) for U * A. We compute the intervention distribution as the joint probability the truncated submodel:
p(S do(A=a) (X x) p(Sa.ba) (X xx)
where p(S.k)
do((Aa)) is the interventional distribution of the causal model (S, and p(Sa.ha) is the joint distri- bution Equation (6) of the truncated submodel (Sa, ba) defined by the intervention. Appendix Figure 11 illustrates the truncation semantics. We next show how the interventional distribution can be used to define a formal semantics for counterfactuals.
2.3 Counterfactual Probabilities
A counterfactual probability = = =x = y) can be read as follows: "Given that obscrved action A = o, and state variables X = followed by outcome Y = y, what is the probability of obtaining an alternative oulcome y', if we were to instead select the action à as an intervention?" Here Ya is a list of potential outcome random variables, distinct from the actual outcomes Y, and X, Y are disjoint. For a given probabilistic SCM = (S,b), we can compute the counterfactual probability as follows (Pearl, 2000, Th.7.1.7):
Abduction/Posteed Update Condition on the observations X a,A= Y y to compute source variable posterior
b'= b(U a. Y y)
Intervention Apply the intervention do(A A a) compute the submodel Sa and the SCM C' (Sa, B/))
Prediction Return the conditional probability P(Y = =y' X = x) computed in the updated SCM:
pC
= a, Y y) p (Y y =
(8)
A posterior update is often called abduction in causal modeling. Through their posterior U, the source variables carry information from the observed configuratinn X = x, A a, Y y to the counterfactual configuration where X x, A = à, Y = y'. Appendix Figure 11 illustrates a counterfactual computation. Counterfactual probabilities form a natural hierarchy as follows.
We refer to the most general counterfactual of the form P(Y, X, A, Y) as a hindsight counterfactual query because specifies the actual outcomes Y.
If the actual outcomes Y are not included in the evidence, have what-if counterfactual query P(Y,|X,A) A) that asks what the likely outcome is after deviating from the actual choice A. For what-if queries, we use the causal effect notation P(Y , A, do Ao(A= = â)) = A,)).
If neither an observed outcome nor an observed action are specified, a counterfactual probability reduces to interventional probability P(Y do A a))-
8

---

Published in Transactions on Machine Learning Research (April/202)
If neither observed outcome nor an intervention specified, counterfactual probability reduces to a conditional probability P(Y|X,A)
Results that we prove for more complez queries immediately hold for simpler queries since they are special cases. For instance, a theorem for what-if probabilities also covers interventional and conditional probabili- ties. In the next section we compare policy evaluation based on conditional and causal probabilitiies.
3 Dynamic Decision Networks for POMDPS
This section describes structural causal model for representing POMDPs. The causal model supports pre- cise semantics for conditional/htteevnniioo hatpr/am///ttrreetittomoomtmtttt probabilities relevant to RL, notably transition and reward probabilities.
POMDP theory and causal concepts share a common formal structure, despite differences in terminology for describing interventions and their effects. Table shows translations between analogous concepts.
Table 4: Correspondence between Causal and RL terminology.
Reinforcement
Causality
Learning
action
treatment.
reward
response
observed slate O
observed co-variales O
state S
co-variates O
belicf state b(5)
latent variable distribution b(Z)
belief state update b(510) complete observability
abduction O()
causal sufficiency
A key difference is that RL concepts are usually defined in terms single state s, whereas causal concepts are defined in terms of values for list of variables. Using the terminology of Russell & Norvig (2010, Ch.2.4.7); RL uses an alomic environment representation, and causal models use a faclored representation, where state is represented factored MDPs are known dynamic influence diagrams (Polich & Gmytrasiewicz, 2007). The basic idea of a dynamic graphical model is to make copy X' for the random variables in a causal graph, to represent successor variables. The dynamic causal graph then a causal graph over the current and successor variables (i.e., over X U X'), such that there are no edges from the successor variables to the current variables.
assignment S to set of state variables S. Graphical models for
3.1 Definition of Dynamic Decision Network
Since unobserved confounders are the key difference between causation and correlation, the need for causal reasoning arises in partially observable MDPs (POMDPs). In order to apply MDP techniques to POMDPs, they are often transformed into a belief MDP. Following Russell & Norvig (2010), we use the term dynamic decision network (DDN) for a dynamic influence diagram that represents belief MDP. Appendix B reviews background on factored POMDPs without reference to causal graphs, including the transformation to belief MDPs. We introduce a new DDN variant for belief MDPs.
Definition 1. dynamic decision network (DDN) for state variables S comprises the following rnndom variables.
1. Current time slice: X S (U (4) U (R) (B)
2. Next time slice: X' - S' u (R' (B').
where resp. B' represent the agent's current resp. successor beliefs.
A DDN D satisfies the following causal assumptions.

---

Published in Transactions on Machine Learning Research (April/202)) 2025)
player health
close to goal
goalie health
player health
close to goal'
goalie health
shoot
scores goal
shoot
scores goal
Figure 4: A dynamic decision network graph for sports scenario. The reward model is indicated by thin lines, and the state transition model by thick lines. Dashed lines represent the agent's 's policy. (The agent's 's belief state is not shown, see text) The line through the Player Health node indicates that in the online version (a), Player Health is observable, and in the offliee version (b), Player Health is not observable.
1. There are no edges from X' to X.
2. There are no edges from R to nodes in X, and edges from R' to nodes in X'.
3. There are no edges from A to nodes in X (R), and no edges from A' to nodes in X' - (R").
These assumptions state that (1) causal relationships respect the temporal ordering. (2) rewards may causally depend on the current state and action, but not vice versa, (3) actions may causally depend on the current state, but not vice versa, and not on the current reward.
De Haan ct al. (2019) argue that Assumption can fail if the temporal resolution of events is low enough that action is assigned the same discrete time index the effect of action, resulting in "causal confusion'. We discuss this scenario further under related work. Our main conclusions do not depend Assumption 3, but use it to simplify formal arguments. Russell & Norvig (2010, Ch.17.4) assune that in DDN, the only parent of an action is the agent's belief state B. However, this assumption does not allow us to represent state variables as direct causes of actions (e.g., the health of a player causes them to shoot).
A fully specified influence diagram, or DDN, defines a joint distribution pP(X, X') over both time slices, as explained Section 2. A DDN therefore specifies an environment MDP as follows.
PP(() is the initial state distribution.
PD(S' A) the state transition model.
(P(RSS,A) is the reward model.
We next give an extended example of a DDN for our sports scenario, which illustrates how a DDN supports causal probabilities for RT..
3.2 Dynamic Decision Network Example
Figure shows the causal graph for a dynamic version of our sports example.
Structural Equations. The graph is parametrized by the structural equations of Table 5, which can be read follows.
For the reward model, a player scores if and only if they shoot, are healthy, close to the goal, and the goalic is not healthy. In the online model Figure 4(i), the player shoots if and only if they are healthy and close to the goal. We treat the player and goalie health as persistent time-invariant features (Lu et al., 2018). Finally for the location dynamics, we make the simplifying assumption that the attacking team stays close the goal if and only they maintain possession. Any state with CG is therefore an absorbing state: CDC 0f CG 0) 1. 12) If the attacking team is close to the goal, our scenario works follows.
10

---

Published in Transactions on Machine Learning Research (April/2025)
Variable
Equation
Shool(
SH = PH CG
Scores( SC)
SC PH CG SH (1 GH)
PlayerHealth' (PH")
PH' = -PH
GoalieHealth (GH))
GH' GH
L,
if CG 0 SH
ClosetoGoal' (onoGGaaal"GG))) (CG")
CG'
Zoo'
if CG 1 and SH 0
Table 5: set of structural equations for the causal graph of Figure 4. Zce denotes a uniformly distributed binary noise variable for CG' (not shown in Figure 4).
Online fig. 4(a)
Offline fig. 4(b)
Conditional
P(SC PPH - CG 1,5H
1/2
P(SC - 1,
1/2
Intervention
P[SC PPH - 1, CG 1, SH1
1) 1/2
P(SC 1 - 1, S0I
- 1/4
What-if
P(SC PH C, 1,SH 0, do SH
1/2
P(SC IPH = 1, 1, 0G= 1,SH 0, SH 1))
Table 6: Scoring probabilities for the reward model in the soccer examples of Figure 4. Appendix F shows the steps for computing these values.
If the player shoots, the team loses possession, either because they scored, or because the shot was blocked and the defending team took over (no rebounds): Pe(CG" 0|SH = 1) = 1.).
If the player does not shoot (e.g., they pass instead), there a 50% change that the attacking team retains possession.
The source variable distribution is uniform:
1/2 PD 1)
PPCCG -)) P 1).
(9)
Figure 4 does not feature belief state variable B, which implies that the agent's decision is Markovian in that it depends only on the current state variables, but not the agent's current beliefs. Markovian policies are also assumed in the theory of confounded MDPs (Zhang & Bareinboim, 2016; Kausik et al., 2024; Bruns- Smith, 2021). Our theorems and analysis address the general case of Mor-Markovian policies that can depend on the agent's entire observation history. Russell & Norvig (2010, Ch.17.4) provide an example DDN with a continnous belief statc.
Example: Causal Reward Probabilities. Table 6 shows the values of conditional and causal reward probabilitics for our example SCM. In the offline model where Player Hcalth is not observed, induces a non-causal correlation between action and reward, which leads to conditional probability of scoring of 1/2, which is higher than the intervention probability 1/4. However, if the common cause between action and reward observed, as in the online model, conditioning on eliminates the non-causal correlation, and the conditional and causal probabilities agree. The next section shows that this is an instance of a general pattern that extends to Q-values: When the causes of the agent's 's action are observable, conditional, interventional, and what-if policy values agrec.
4 Policy Evaluation for Belief MDPs
The policy evaluation task compute the value function for policy क in given environment. In typical RI setting, evaluate a policy learned from data generated by behavioral policy æs; see Figure 2.2
The learned policy called the estimation policy in (Sutton & Barto, 1998, Ch.5.6), target policy by Wan et al. (2021). and the evaluation policy by Bruns-Smith (2021).
11

---

Published in Transactions on Machine Learning Research (April/2025)
In factored POMDP, the agent's decisions depend not only their current belief state, but also their current observation. Accordingly, an epistemic state (o.b) b) comprises a current observation and a current belief state. We use the tuple notation () to make longer formulas easier to parse, and to emphasize the analogy between epistemic states and MDP states in a traditional atomic representation. A policy maps an epistemic state to a distribution over actions:
T O B" ((A)
where O" represents the observation space of the agent executing policy a and D" the space of belief states, ie. distributions A(Z*) over the unobserved variables. We also use the conditional probability notation b))
Since a DDN defines environment process, can use it to perform model-based evaluation of policy, by computing the required reward and transition probabilities from the DDN. We focus on what-if reward and transition probabilities, since they cover interventional and observational probabilities as a special case. In RL based on what-if counterfactual decisions, the learning agent seeks to learn an optimal policy that may deviate from the actual decisions laken. Following Pearl's suggestion in a similar context (Pearl, 2000, Ch.4.1.1); we refer to observed decisions by the behavioral agent as "acts", with associated random variable A and active decisions by the learning agent "actions", with associated random variable A.
4.1 Bellman Equation for Causal Q-functions
Given a policy x, the Bellman equation for the what-if Q-function as follows.
(), R a, do(a)) + pold P(.
(10)
VT, Σ
do((a)
(11)
RP a b), do(a)) = Ex do(a)) Σr. Po = r/z, o, do(a))
where E [0.1] is a discount factor. In the equation shown, the epistemic states <o, a, b) are defined by the policy's observation space O The pD terms are defined by the DDN as shown in Section 3. As usual the state value V is the expectation of Q-values over actions the agent may take in the state (eq. (11)). Equation (10) finds the current Q-value by computing the current expected reward and the expected value of the state reached after the next observation o'.
According to recurrent Equation (10), given a new observation of and the current action a, the expected policy value can be computed in two steps:
Posterior Update Compute the new belief state b' by conditioning the current beliefs b on observations o, o' and action a. The posterior belief D' can be computed by the well-known belief update formula, which we derive for factored POMDP in Appendix B.2 (Equation (16)).
Prediction Estimate the expected return V (='('', given the new observation o' and new belief state b',
Updating a posterior to predict the outcome of an action is key part of the formal semantics of counter- factuals that we presented in Section 2. Appendix G illustrates how the different Bellman equations can be used to evaluate a policy. Bellman equations for the interventional Q-function Q b), do(ā)) and the traditional observational Q-function Q b), a) are derived as special cases of Equation (10). Appendix C writes them out and shows how the interventional and what-if Q-functions are straightforwardly obtained from the standard observational Bellman equation. Policy evaluation based on the interventional Q-function has been studied in previous work (Wang et al., 2021; Zhang & Bareinboim, 2020).
12

---

Published in Transactions on Machine Learning Research (April/2025)
4.2 Causal versus Conditional Policy Evaluation
Our main policy evaluation result states that if the causes of the action variables are observable, then what-if Q-values, interventional Q-values, and conditional Q-values are equivalent. A technical issue is that in an SCM, cach variable may have an unobserved noisc variable as a cause. Instead of complete observability, the causal modeling literature therefore utilizes a more general causal sufficiency condition that allows a variable have a latent cause, but not shared latent cause (Spirtes et al., 2000), that is, confounder. Formally we say that source variable U is noise variable for X if U is independent of every variable except X. In Figure 4, goalic health is noise variable for scoring. A set of observed variables O is causally sufficient. for variable X in causal graph G if every latent parent of X is noise variable for X. An SCM C' is action sufficient if its graph is causally sufficient for action variable A.3 The next proposition states our main result.
Theorem 1. Suppose that an observation signal O is action sufficient in dynamic probabilistic SCM D. Then for every epislemic stale, observed act a, and action à:
(,, a, b), do(ā)) = Q (k, do(ā)) = QT. ((o
That is, given that the direct causes of on act have been observed, the Q-value associated with an intervention à replacing the actual act equals the Q-value associaled with condilioning on å, independent. of the acl.
The upshot is that under action sufficiency, what-if and interventional Q-values can be eveluated using conditional probabilities, as is done in traditional RL. The proof of Theorem 1 is given in Appendix D.1. An informal outline as follows: We show as a general lemma in causal theory that the parent values of manipulated variable are given, what-if probabilities intervening on A are equivalent to conaltional probabilities observing A. This result can be used show that what-if trraadttraaniiion probabilities are equivalent to observational recursively in lerms of reward/transiie probabilities, the equivalence of swwrdd/rannition probabilities implies the equivalence of the Q-functions.
/transiiion probabilities. Since the Bellman equations define Q-values
Remarks Although Theorem 1 is stated in terms of a causal model, it does not assume that. the learning agent is given a true causal model of the behavioral policy and the environmental process. Rather, the import that as long as the learning agent's observation signal includes the causes of the behavioral agent's 's actions, conditional probabilities inferred from the action data are equivalent to interventional probabilities, matter what the true dynamic causal model is.
Theorem 1 does not cover hindsight Q-values. Indeed remarkable feature of counterfactual hindsight Q-values is that they can differ from conditional probubilities even under action-suffciienen Appendix E provides worked-out counterexamples
Examples Table 7 illustrates Theorem 1. The Q-values are similar to the reward values in Table 6, so we show the state values, which capture the sequential dynamics of Bellman's equation. In the on-line model
3Our concept of action-sufficiency differs from the notion introduced by Huang al. (2022). For them, latent state is action sufficient is powerful enough support an optimal policy based on the latent state space. Our concept could be called "causal action sufficiency" to disambiguate.
Online Figure
Offline Figure 4
Conditional
VP. 1, CG = 1,b) = 1/2
Va. CG = 1,b) = 1/3
Intervention
VPH = CG = 1,6) 1/2
V.. (CG = 1, = 1/6
do
What-if
Ni DPH 1, CG 1H 0) 1/2
VI. 1,b)
Table 7: Value functions in the soccer examples of Figure 4. In the online version of Figure 4, Player Health is observable, in the offinn version it is not. The belief b the uniform prior over source variables. Appendix G shows the steps for computing the offline model values. The notation Vao denotes value function that averages over next step interventional Q-values (Equation (10)).
13

---

Published in Transactions on Machine Learning Research (April/2025)
of Figure 4, where Player Health is observable, evaluate the standard behavioral policy (shoot if and only if close to the goal and healthy). Since this policy shoots deterministically when close to the goal, the state values agree with the reward probabilities of Table 6.
For the offiine model of Figure 4, where Player Health is not observable, we evaluate the marginal policy I derived from the behavioral policy by averaging over latent states. The marginal policy is an important concept in offline policy evaluation (Kausik et al., 2024; Bruns-Smith, 2021). It can be viewed as a baseline form of behavioral cloning that estimates the agent's action probability from frequencies based the obser- vation signals. In example, if the player is close to the goal with unknown health status, the probability that they take a shot is 1/2. So we have the following standard marginal policy for our sports example:
1/2, if CG 0,
P(SH = CG)
(12)
otherwise
Observe in Table 7 that in the non-executable offline setting, the marginal policy has different values de- pending on whether it is evaluated based on conditional, interventional, what-if probabilities.
5 Action Sufficiency in RL Settings
This section connects the action sufficiency condition of Theorem 1 to policy executability and reviews the importance of our theoretical analysis for common RL settings.
A policy π executable if ooaKo, = T(aKo, b), =) for any latent state variable z that is not a noise variable for A. Executability equivalent to the act-state independence condition discussed by Gasse et al. (2021). Intuitivcly, exccutability guarantoes that the policy obscrvation space contains sufficient information for assigning probabilities to decisions in accordance with the policy.
Just DDN defines conditional probabilities that model an environmental process, it defines conditional probabilities that model behavioral policy. We say that a DDN matches a behavioral policy те if o Ko, b)) = (alo,, b) for all actions a and beliefs b. The next observation states that a policy matched
by a DNN is executable if and only if the DNN is action sufficient. requires a minor technical definition that rules out redundant parents: We say that a CBN action-ménimal for every parent X of action variable and every set V of variables disjoint from and X, we have P(A[X, V) # P(A[V). That is, there is variable set V such that conditioning V makes X independent of its child A. Local minimality is entailed by the well-known stability Salllittyyfiiti /fuites conditions (Pearl, 2000, Ch.2.4).
Obscrvation 1. Let D be locally minimal dynamic decision network that matches behavioral policy ng. Then Te is executable if and only if the policy obscrvation space O is action sufficient in D.
Note that Observation 1 does not assume that DDN matching Tp is given. The import is that, long behavioral policy is executable, it can be represented by action sufficient DDN. Therefore for executable behavioral policies, Theorem applies, and what-if if probabilities can be computed from conditional probabilities.
RL with executability. As we explained in the introduction (Table 2), various common RL settings satisly executability, so we may summarize our argument as follows:
On- policy learning OR online exploration OR complete observability executability conditional policy values = what-if policy values interventional probability policy values
where online exploration refers to any setting where the learning agent uses the behavioral policy to generate observations.
RL without executability. When none of the sufficient conditions hold, we have the partially observable offline off-policy (PO00) setting. In the POOO selling, policy evaluation based on conditional probabiláties
14

---

Published in Transactions on Machine Learning Research (April/2025)
can, and typically does, lead to wrong policy values, even in the asymptotic setting with perfect probability estimates. To highlight this important point, we discuss three examples in an informal way to convey the intuition for how confounders mislead policy estimation in the POOO setting.
Scenario 1: Sports. Consider the offiin scenario of Figure 1b.
Vancouver Canucks coach Rick Tocchet watches the Edmonton Oilers to learn from the season finalists. Based on a very large sample of games, he notices that whenever the Oilers shoot close to the goal, they score 50% of the time. So he directs the Canucks players to shoot whenever they gel close. Tocchet is disappointed to find that the Canucks score only 25% of the time. "It must be that my players are worse than theirs", he thinks. Question: Is the coach right blame his players?
The answer is no: Because Tocchet did not observe the health of the Oilers players, he did not realize that they shoot only when they are healthy. His policy directs the Canucks players to shoot whether they are healthy or not, which leads to a lower success rate. In this example, using conditional probabilities leads to an overestimate of the Q-value of shooting. The conditional probability is 1/2 in the model (cf. Table 6). The correct Q-value estimate is given by the interventional probability of scoring given the intervention shoot, which is 1/4: Given that the intervention breaks the link between shooting and goalie health, the interventional probability is the joint probability that the player is healthy and that the goalic is not, which is 1/2 1/2.
Scenario 2: Imitating Expert Drivers. Consider a simplified version of the driving scenario of Zhang et al. (2020), where the expert driver brakes if and only if the car in front turns on its tail light (cf. Figure 5). In their scenario, the lail light is observable to the demonstrator but not to the imitator. Suppose that the real- world frequency of observing the tail light is 25%. The marginal behavioral policy then randomly brakes 25% of the time. Conditional probabilities misestimate the value of the marginal policy follows.
Since the expert hardly ever suffers an accident after not braking, the conditional Q-value of not braking is high. But the braking decisions of the imitator are independent of the tail lights of the car in front, so the probability of an accident after not braking is a substantial 25%, which is the interventional Q-value. Thus the conditional Q-value overestimates the interventional Q-value.
When the imitator brakes, but the car in front does not, they slow down unnecessarily. Assuming the reward function includes the time to destination, the conditional Q-value too low compared to the interventional Q-value.
Scenario 3: The Frozen Lake. Warrington et al. (2021) describe imitation learning in frozen lake scenario where the expert has access to image data that show the locations of weak ice, but the trainee does not. In their simulations, the expert agent casily learns to cross the lake safely by taking into account where the ice is weak. The trainee observes that the expert never falls into the lake, and wrongly concludes that crossing lake is a safe policy. In policy evaluation terms, they overestimate the value of crossing the lake compared to the policy of walking around it. In the simulations of Warrington al. (2021), this policy mis-estimation leads to catastrophically poor return for a naive imitation learning algorithm that ignores confounding.
6 Related Work: Current and New Research Directions
This section selectively describes current RL research, and some future directions, involving causal models with an emphasis on the distinction between observational, interventional, and counterfactual probabilities. We describe several directions for future research from the perspective of confounding and executability. For comprehensive surveys of causal RI, please see Bareinboim (2020), (Deng et. al., 2023). Bareinboin et al. (2025) and (Schölkopf et al., 2021, Section E).
A common goal in previous research leveraging given causal model. Such approaches can be categorized causal model-based R.L. Causal model-based RL inherits the challenges and benefits of model-based RT.
15

---

Published in Transactions on Machine Learning Research (April/2025) 2025)
front tail light
car front
brake
accident
own brake light
Figure 5: A car-driving scenario which the state variable brake light is an effect of the braking action.
in general (Levine et al., 2020, Scc.5). Our discussion focuses on the following special features of causal model-based RL:
1. The ability to correctly evaluate the effects of interventions when confounders are present.
2. The greater expressive power of causal models, which define not only conditional reward-transition models, but also interventional and counterfactual reward-transition models.
3.
The causal graph structure, which decomposes joint distributions into local mechanisms.
6.1 Online Causal RL
The term online RL in this section refers to single-agent online exploration, where the learning agent is the only agent exccuting behavioral strategy to learn about the environment. Our analysis so far implics that in single-agent online learning, the importance of advantage 1. However, the last two advantages (graphical structure and greater expressive power) apply to online learning as well, and have been leveraged in previous work on online RL.
expect conditional probabilities to be unconfounded, which reduces
Causal Graphs and State Abstraction: Eliminating Irrelevant Variables Lemma 1 implies that conditioning on superset X 2 A of observable causes (parents) of A suffices to ensure that conditional probabilities are causal. This means that the conditioning set need not include all observed variables, that is, it need not include the entire state observed by the agent. For example in the car driving model of Figure 5, for the braking decision, it suffices to condition on FroniTaillight and CarinFront; the OwnBrakk Variable can be ignored during the decision process. In general, a state variable S conditionally irrelevant a decision if citpanctioo/mmmpfffctatiii /simpiification (Peters et al., 2017, Sec.8.2.2). The paper by Sen et al. (2017) is one of the first to leverage given causal graph to reduce the effective state space. They prove that this reduction improves regret bounds in online bandit problems, when the bound is a function of the size of the state space. Zhang & Bareinboim (2020) provide an algorithm for reducing the state space by eliminating irrelevant variables given a causal model, which leads to substantive improvement in regret bounds. Wang et al. (2022; 2024) relate the elimination irrelevant. variables from a dynamic causal model to state abstraction.
R, S', PaA. Thus a causal model supports variable selection as a form of state
abstraction
Causal Graphs and Imitation Learning The goal of behavioral eloning is to learn a policy that is similar to that of the behavior policy that generated the data. Using our notation, the goal is to learn a policy и such that b(A[S)= x π (AIS) where is the behavior policy to imitated. De Haan et al. (2019) in a paper on "causal confusion" point out that the conditional probability behavioral cloning objective can be problematic, even assuming exccutability. For example, in self-driving car example like that of Figure 5, the observation signal includes the driver's own brake light. Since the brake light comes on only when the driver brakes, we have a very high correlation between braking and the brake light being on:
P( ((Brake
1 BrakeLight 1) 1 and P Brake 1 BrakeTight 0)
16

---

Published in Transactions on Machine Learning Research (April/2025)
Therefore if the imitator matches conditional probabilities according to this criterion, they will brake only if they observe the brake light coming on. In this scenario, the imitator will not respond to the location of the other cars and their tail lights, which fails to match the expert driver's behavior (and fails to avoid accidents). With knowledge of the cansal graph Figure 5, we can climinate the irrelevant brake light variable, as described in the previous paragraph, and avoid the causal confusion problem.
Zhang et al. (2020) introduce a different imitation objective, matching the return of a demonstrator. They introduce the concept of different observation spaces for the imitator and demonstrator, and state necessary and sufficient graphical conditions on given causal graph for when imitating policy match the demonstrator's return, even when the observation signals of the imitator and demonstrator are different.
Data Augmentation and Hindsight Counterfactuals One of the traditional usee of models in RL, going back to the classic Dyna system (Sutton, 1990), is to augment the observed transition data with virtual experiences simulated from the model. Sun et al. (2024) utilize hindsight counterfactuals to generate virtual state transitions that specify the next state that would have occurred in the same scenario. These counterfactual state transitions take the form P(S A [S, S", where we observe a next state transition from S' to S' and ask what the next state would have been if the agent had selected action Á instead of A.
Compared to traditional state-transition models of the form S, A), hindsight counterfactuals condition on more information and thus are polentially more accurate in generating virtual transitions. Sun et al. (2024) provide empirical evidence that hindsight state transitions speed up learning an optimal policy. Generating hindsight state transitions requires a causal model and is not possible with a traditional RL transition model that is based on conditional probabilities only.
Learning A Causal Model from Online Data Causal model discovery methods that are applicable to online RL learning include deep models based auto-encoders (Lu et al., 2018) and GANs (Sun et al., 2024). Such deep generative models generate observations x from latent variables (cf. Equation (2)), but they are not structural causal models based on a causal graph that represents local causal mechanisms. Huang et al. (2022) show how constraints from a given causal graph can be leveraged to learn latent state representations. Learning an influence diagram over state variables from online RL data seemss to be a new research topic. An exciting new possibility for online learning is that the agent's 's exploration can include experimentation in order to ascertain the causal structure among the state variables.
6.2 Offline Causal RL
The off-policy evaluation (OPE) problem is to estimate the value function of a learned policy that is different from the behavioral policy. OPE is one of the major approaches to offline RI., where information about the behavioral policy is recorded in a previously collected dataset (Levine et al., 2020). The examples in this paper illustrated OPE based a causal model (Section 4). In settings without executability, interventional policy evaluation is different from conditional policy evaluation, and research has therefore focused on such settings.
Leveraging a Causal Model for OPE Wang et al. (2021) investigate policy optimization based on interventional probabilities. To estimate the interventional probabilities when the behavioral policy is not executable, they assume that a causal model over the entire state space is available. The problem is then to compute a marginal interventional probability such as P(SC)dol (SH), CG) in our running example, from conditional probabilities over the entire state space, such as P(SCISH, CG, PH). Pearl's do-calculus provides powerful techniques for inferring marginal interventional probabilities from conditional probabilities, using what are known adjustment formulas. Two well-known types of adjustment formula are the backdoor and the frontdoor critcrion. Wang et al. (2021) utilize both to compute causal reward and state-transition probabilities from a given causal model, and show how to use the causal probabilities in an intervent ional Bellman equation. To illustrate the idea in our sports example, consider offline learning in the confounded model of Figure 1b. Since in this setting, only closeness-to- to-goal observable, an exccutable policy would be based on this variable only (e.g. "shoot whenever you are close to the goal"). Finding interventional values for such a policy involves computing interventional probabilities such as P(SC C|CG 1, do(SH 1)).
17

---

Published in Transactions on Machine Learning Research (April/2025)
According to the backdoor criterion, such probabilities can be computed by marginalizing over the unobserved values of player health as follows:
P(SC =
= 1, =1)
P(PH = 1)P(SC 1PPH = - 1, 1 - 1|PH- = 00, CG = 1,5H = =1))
= =12 1/2- = -1/4
which agrees with the result of Table 6. While adjustment formulas provide an elegant approach to addressing spurious correlations in OPE, is not entirely clear what their OPE use case is, as the learning agent does not have access to the latent variables that appear in the adjustment formulas. A possibility is that the behavioral agent uses their access to latent variables (e.g., the athlete has access to their health) to compute marginal interventional probabilities and communicate them to the offline learner.
Causal OPE Most work is based on confounded MDPs (Zhang & Bareinboim, 2016; Kausik et al., 2024; Bruns-Smith, 2021): States are decomposed into an observed part and an unobserved part. The behavioral policy is assumed to depend on the complete state, whereas the policy to be evaluated depends on the observation signal only. A current line of research gives bounds on the extent of the bias cue to spurious correlations, based on assumptions about the confounders (Kansik et al., 2024; Bruns-Smith, 2021). For example, "memoryless confounders" are sampled independently at each time instant. At the other extreme, time-invariant latent variables such as Goalie Health or Player Health in our example are known "global confounders" (Kausik et al., 2024). Some of the causal OPE methods are model-based in the sense of estimating transition probabilities, but not in the sense of utilizing dynamic SCM. Another difference is that the evaluation policies considered are Markovian in that they depend on current observations only (cf. Section 3.2). In contrast, our belief MDP framework allows for evaluation policies that depend on past obsorvations/ ccurrntt belicfs.
Warrington et al. (2021) and Ghosh et al. (2021) propose elfective Bayesian approaches to confounding, where the learning agent maintains a distribution over information that was used by the behavioral agent during training, but cannot be directly observed by the learning agent at test time (e.g., location of ice patches, target locations map). They do not use the confounded MDP framework. A valuable topic for future research is whether causal modeling would further improve the Bayesian approach.
Learning Causal Model from Offline Data One possibility is to learn fully specified structural causal model with latent variables and use it for interventional policy evaluation (Lu et al., 2018). For learning a causal model offfine, it is likely that the extensive work on learning Bayesian networks for temporal data can be leveraged, including recent. approaches based on deep learning and continuous optimization (Sun et al., 2023). Latent variable models related deep generative models are a promising direction for learning structural causal models (Geffner et al., 2022; Sun & Schulte, 2023; Mooij et al., 2016; Hyvärinen et al., 2010). Much of the research learning a causal model assumes causal sufficiency (no confounder),, but not all of it (Schölkopf et al., 2021). Learning an influence diagram with causal graph over state variables from offline RL data scems to be a new rescarch topic.
6.3 Offline + Online RL
In this hybrid setting, the learning agent interacts with the environment to collect more data, but a prior dataset is also utilized (Sutton, 1990; Janner et al., 2019). We can view this as an off-policy setting (see Fig- ure 2a). A successful example is the AlphaGo system, which used an offline dataset of master games to find a good initial policy through imitation learning, then fine-tuned the policy with self-play (Silver et al., 2016). An active topic of research is how causal models can leverage oflline datasets for online learning (Gasse et al., 2021; Zhang& & Bareinboim, 2020), sometimes called generalized policy learning (Bareinboim, 2020).
Off-policy model-based RL approaccees (Levine et al., 2020, Sec.5.2) can be applied with causal models leverage the offline dataset. The techniques outlined for causal models in the online and offline settings
18

---

Published in Transactions on Machine Learning Research (April/2025)
can be utilized in the hybrid offine//nline setting well. For example, can learn causal graph from the offline dataset, and fine-tune it during online interactions with the environment. The ability to intervene in an environment in the online setting is potentially a powerful tool for learning a dynamic influence model. For instance, performing experiments can resolve the causal direction between two variables which may not be possible from observational data alone, at least not without assuming causal sufficiency.
Zhang & Bareinboim (2020) describe hybrid approach for joint exploration and policy learning that is not based model. The offiine dataset is used to estimate conditional state-reward transition probabilities. These estimates may not be correct interventional probabilities if confounders present. Causal inference theory has established theoretical bounds on how far a conditional probability can differ from the interven- tional probability. Zhang and Barenboim use the resulting interval estimates for interventional probabilities as an input to optimistic exploration for online learning, where policies are evaluated according to their max- imum possible value. Optimism under uncertainty is well-known approach in RI. for ensuring extensive exploration of the state space (Osband & Van Roy, 2014). The offline dataset provides tighter bounds on interventional probabilities than learning from online data only, thereby speeding up optimistic exploration.
7 Summary and Conclusion
We believe that many RL researchers share the intuition that in common RL settings, conditional probabili- ties correctly estimate the causal effects of actions. Our paper spelled out the conditions where we can expect conditional probabilities to correctly measure causal effects, defined by interventional probabilities. We pro- vided rigorous argument, using the formal semantics of causal models, for why these conditions lead to correct estimates. The key condition is executability; the observations available to the learning agent include the obscrvations available to the behavioral agent, such that the Iearner can exccute the behavioral policy. Our argument was that it is only in partially observable environments with offline off-policy learning (the POOO setting) that executability fails, and conditional probabilities can diverge from observed conditional probabilities. The reason is that in this learning setting, the environment may contain confounders, not observed by the learning agent, that infiuence both the decisions of the behavioral agent and the states and rewards that follow these decisions. Such confounders can introduce spurious correlations between decisions and states/reware that do not correctly estimate the causal impact of the agent's decisions.
In contrast, if an RL setting satisfies executability, conditional probabilities coincide with interventional probabilities. Our argument for this conclusion involves two steps. (1) In such environments, the set of variables that the learning agent can observe is causally sufficient for the behavioral agent's actions, in the sense that it includes all common causes of the behavioral agent's 's decisions and other variables. (2) We prove formally, using causal theory, that if a set of observable variables is causally sufficient for actions, then actions are not confounded with states or rewards. Thus our analysis relics on the distinction between the observation signals available to the learning and behavioral agents, which has been highlighted by previous work in causal RT. (Zhang et al., 2020; Zhang& & Bareinboim, 2016; Kausik et al., 2024; Gasse et al., 2021).
In addition to the causal effects of interventions, causal models provide rigorous specification of coun- terfactual probabilities through formal semantics. Causality researchers have recently proposed using counterfactuals to enhance reinforcement learning (Bareinboim, 2020; Deng et al., 2023). We therefore ex- tended our analysis to counterfactuals distinguishing two kinds of counterfactuals: what-if queries (e.g. if choose action d' in state s instead of action a, what is the likely reward?) and hindsight counterfactuals that condition on an observed outcome (e.g. given that I received reward r after choosing action a in state 8, what is the likely reward if I choose action a' instcad?). We showod that under exccutability, what-if queries can be correctly estimated from conditional probabilities, but hindsight counterfactuals go beyond conditional probabilities (cf. Sun et al. (2024)).
Based on our analysis, we discussed the potential benefits of causal models in different reinforcement learning settings, such as online, offline, and off-policy. The most forward though not the only, approach is to follow a model-based RL framework, replacing the traditional models involving conditional probabilities with a causal model. Structural causal models offer three main benefits: (1) They distinguish interventional and conditional probabilities, and therefore causation from correlation. (2) They factor the dynamics of a complex environment into local causal mechanisms represented in causal graph. Local mechanisms are typically
19

---

Published in Transactions on Machine Learning Research (April/2025)
invariant under interventions (Schölkopf et al., 2021), which means that causal graph can help address the challenge of distribution shift. (Tevine et al., 2020), where a learned policy is evaluated against data gathered by another policy. (3) Causal models have greater expressive power than conditional probability models, since they can also evaluate interventional and counterfactual probabilitios. We described existing work and promising urture directions for how the benefits of causal models can be leveraged for reinforcement learning.
Reinforcement learning and causality are areas of Al and machine learning that naturally complement each other. The analysis in this paper provides guide for reinforcement learning rescarchers as to when and how they can make use of causal concepts and techniques to advance reinforcement learning.
Acknowledgements
We are indebted to the anomymous reviewers for very helpful suggestions. This research was supported by discovery grants to Poupart and Schulte from the Natural Sciences and Engineering Research Council of Canada.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, NIPS'17, pp. 5055-5065, 2017.
Elias Bareinboim. Towards causal reinforcement learning. International Conference Machine Learning Tutorial, 2020. URL https://crl. causalai. net.
Elias Bareinboim, Junzhe Zhang, and Shanghack Lee. An introduction to causal reinforcement learning. Tochnical report, Causal Artificial Intelligence Lab, Columbia University, 2025.
David A Bruns-Smith. Model-free and model-based policy evaluation when causality is uncertain. In Inler- national Conference on Machine Learning, pp. 1116-1126. PMLR, 2021.
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
Zhihong Deng, Jing Jiang, Guodong Long, and Chengqi Zhang. Causal reinforcement learning: A survey. arXiv preprint arXio:2307.01422 2023.
Maxime Gasse, Damien Grasset, Guillaune Gaudron, and Pierre- Yves Oudeyer. Causal reinforcement learn- ing using observational and interventional data. arXio preprint arXiv:2106.1444 2021.
Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. orXiv preprint ar arXiv:220e.021 2022.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why gener- alization in RL is difficult: Epistemic POMDPs and implicit partial observability. Advances in neural information processing systems, 34:25502- 14-25002-2555,, 2021.
Schoon Ha, Joonho Loe, Michiel van de Panne, Zhaoming Xie, Wenhao Yu, and Majid Khadiv. Learning- based legged locomotion; state of the art and future perspectives. arXiv preprint arXiv:2406.01 01152, 2024.
Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Heess, Hado P van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup, et al. Hindsight credit assignment. Advances in neural information processing systems, 32, 2019.
Matthew Hausknecht and Peter Stone. Deep recurrent Q-learning for partially observable mdps. In 2015 AAAT Fall Symposium Series, 2015.
20

---

Published in Transactions on Machine Learning Research (April/2025)
Biwei Huang, Chaochao Lu, Liu Leqi, José Miguel Hernández-Lobate, Clark Glymour, Bernhard Schölkopf, and Kun Zhang. Action-sufficien state representation ation learning for control with structural constraints. In International Conference on Machine Learning, pp. 9260 9279. PMLR, 2022.
Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of structural vector autoregression model using non-Gaussinnity. Journal of Machine Learning Reseorch, 11(5), 2010.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-basod policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Leslie Pack Kaelbling, Michael L. Littman, and Authony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 101(1-2):99 (11-2)9999199 13:,, may 1998. ISSN 0004-3702.
Chinmaya Kausik, Yangyi Lu, Kevin Tan, Maggie Makar, Yixin Wang, and Ambuj Tewari. Offliee policy evaluation and optimization under confounding. In International Conference on Artificial Intelligence and Statistics, pp. 1459-1467. PMLR, 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXie preprint 0rXiv:2005.06443 2020.
Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández-Lobato Deconfounding reinforcement learn- ing in observational settings. arXiv preprinl or Xin: 18x:1111... 10576, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, el al. Human-level control through deep reinforcement learning. Nature, 518(7540):522) 533, 2015.
Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using obsorvational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1):1103 1204, 2016.
lan Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored MDPs. Advances in Newral Information Processing Systems, 27, 2014.
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge university press, 2000.
Jonas Peters, Dominik Janzing, and Beruhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Kyle Polich and Piotr Gmytrasiewicz. Interactive dynamic influence diagrams. In Proceedings of the 6th international joint conference Autonomous agents and multiagent systems, pp. 1-3, 2007.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Procoedings, 2011.
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 2010.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612 634, 2021.
Rajat Sen, Karthikeyan Shanmugam, Alexandros Dimakis, and Sanjay Shakkottai. Identifying best in- terventions through online importance sampling. In International Conference on Machine Learning, pp. 3057-3066. PMLR, 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Dricssche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of with deep neural networks and tree search. nature, 529(7588):484 2016.
P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT Press, 2000.
21

---

Published in Transactions on Machine Learning Research (April/202)
Xiangyu Sun and Oliver Schulte. Cause-effect inference in location-scale noise models: Maximum likelihood v8. independence testing. In Advances in Newral Information Processing Systems, 2023.
Xiangyu Sun, Oliver Schulte, Guiliang Liu, and Pascal Poupart. NTS-NOTEARS: Learning nonparametric DBNs with prior knowledge. In International Conference on Artificial Intelligence and Statistics. PMLR, 2023.
Yuewen Sun, Erli Wang, Biwei Huang, Chaochao Lu, Lu Feng, Changyin Sun, and Kun Zhang. ACAMDA: improving data efficiency in reinforcement learning through guided counterfactual data augmentation. In AAAI, pp. 15193-15201. AAAT Press, 2024.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dy- namic programming. In International Conference Machine Learning, pp. 216-224. Morgan Kaufmann, 1990.
Richard S Sutton and Andrew G Barlo. Introduction to reinforcement learning. MIT Press Cambridge, 1998.
Yi Wan, Abhishek Naik, and Richard S Sutton. Learning and planning in average-reward Markov decision processes. In International Conference Machine Learning, pp- 10653-10662 PMT.R, 2021.
Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems, 34:21164-21175, 2021.
Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task- independent state abstraction. In International Conference Machine Learning, July 2022.
Zizhao Wang, Caroline Wang, Xuesu Xiao, Yuke Zhu, and Peter Stone. Building minimal and reusable causal state abstractions for reinforcement learning. In Proceedings AAAI, 2024.
Andrew Warrington, Jonathan W Lavington, Adam Scibior, Mark Schmidt, and Frank Wood. Robust asymmetric learning in pomdps. In International Conference Machine Learning, pp. 11013-11023. PMLR, 2021.
Junzhe Zhang and Elias Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Purdue AT Tab, West Lafayette, IN, USA, Tech. Rep, 2016.
Junzhe Zhang and Elias Bareinboim. Designing optimal dynamic treatment regimes: A causal reinforcement learning approach. In International Conference on Machine Learning, pp. 11012-11022. PMLR, 2020.
Junzhe Zhang, Daniel Kumor, and Elias Barcinboim. Causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 33:12263 12274, 2020.
A Online and Off-Policy Reinforcement Learning
In online reinforcement learning, the agent interacts with their environment through a behavioral policy to gather data. In the on-policy setting, the learned policy is evaluated using data generated by the learned policy itself. Figure 6 illustrates these learning settings with a diagram.
B Background: Markov Decision Processes
For temporal data the difference between causation and correlation stems from the possible presence of unobserved common causes of the agent's actions and other environment variables. The RL setting in which parts of the environment may be unobserved, which is known as partially observable Markov decision process (POMDP). This section reviews the basic theory of POMDPs with respect new factored POMDP variant that facilitates applying causal concepts.
22

---

Published in Transactions on Machine Learning Research Aprri/20255
rollout data ([8;, B4, S/,, r.))
rollout data ((8,, a;, 8,p (t))
8,
Update
k
8, T
Buffer
a
D
k
rollout(()
a
Update
rollout(s)
kh++1
(a) Online RT.
(b) Off-policy RT.
Figure 6: Online RL settings; figures adapted from Levine et al. (2020). Figure 6a: In classic online RL, the policy Te is updated with streaming data collected by H itself. Figure 6b: In classic off- off-policy RL, the agent's 's online experience is appended to data buller (also called replay buffer) D, each new policy me is used by the agent to collect additional data, such that D comprises samples from To...., Tk: and all of this data is used to train an updated new policy T++1- In both online settings the behavioral policy is ezecutable, because the policies used to generate the data are based on the same observations as the learned policy.
B.1 Factored Partially Observable Markov Decision Processes
The state space of factored POMDP is defined by finite set of state variables S. An environment state is an assignment s to the state variables. The state variables are partitioned as
S=
where represents the set of observable state variables, also called the observation signal, and repre- sents the latent state variables, which we sometimes refer to simply as the latent state. Z is empty, the state is completely observable; otherwise it is partially observable. Since pair (z, describes state, freely apply MDP notation to both and pairs (z, o). In addition, have action variable A ranging over finite set of actions available to the agent, and real-valued reward variable R.
A POMDP comprises the following components.
An initial state distribution Pε(so), which can be factored into a distribution over latent and ob- served variables:
Pr(so) = Pr(zo, oo) = Pr(za) x PF(On|Z0)
where Pr(2o) is an initial latent distribution and (2)(0)))) is the initial observation model.
A transition model, which factors into latent state update model and dynamic observation model (see Figure 7a):
PE($141|9,, az) = 0-1ls, ac) Pe( ac) Pg Or, ar).
We assume that the current observations are Markovian in that they depend only on the current latent state, and the most recent observation and action:
Pe(Or+1|zet1. Ot, ax) Pe 1
[z+1, Ogt, age)
(13)
23

---

Published in Transactions on Machine Learning Research (April/202)) (025)
Ze+1
Z1
A
d
O++1
0
b++1
b++1
A++1
A
O+11
(a) Transition POMDP.
model
factored (b) Transition model and executable policy in belief MDP.
Figure 7: Generic causal graphs for POMDPs. Dashed lines represent generic policy. Figure 7a: The environment transitions to new latent state and generates an observation signal based on the current state and most recent action. Figure 7b: The agent selects an action through their policy based on their current belief state b. Their updated belief state depends on the previous belief state, previous action and new observation.
A policy π (or agent function Russell & Norvig (2010)) maps a finite sequence of observations and actions distribution P(A) e A(A) over actions:
T(0e/o0, do 0n, 01 o.) maalor act, 0ct)-
A policy is executable if actions are independent of latent states, such that for all times we have
T(ae|ze, Oge, azt)
(14)
B.2 Posterior Updates
Given a current action a, and observation Ot+1, the agent's beliefs move from current beliefs b, to updated beliefs de+1 through posterior updates; see Figure 7b. The observation signal provides agent with infor- mation about the latent environment state through the posterior distribution P(z+-100441, ag)) where z is the current latent state, 0gt+1 is the current observation sequence, and age is the previous observation sequence before taking the current decision. We derive the well-known POMDP formula for recursively updating the latent state posterior (Russell & Norvig, 2010, Ch.17.4.2) for a factored POMDP and an exe- cutable policy. For the purposes of the derivation, we define executability in terms of observation sequences in Equation (14), rather than the epistemic state notion of Section 5, which provides more general result.
Observation 2. If the transition data are generated by POMDP Pg and an executable policy, the latent posterior updale is given by
Plz++100ett az)) a Oz, oz) x Σ Pp(ze41 or, (z) acs)
(15)
2,
where a a normalization constant.
Proof.
Peeelleenneeeeess/////1k11 Σ Pa(z41l ogn Ozt
Pzzz age) by Bayes' theorem
P Oz, a:) Σ PE(24112 Pl) ase) applying the Markov property Equation (13)
PE(or o,, Σ
P Or, (( aze) because is executable
24

---

Published in Transactions on Machine Learning Research (April/2025) 2025)
The Belief MDP Even if the environment dynamics is Markovian in the state space, may not be Markovian in observation space, because past observations can and typically do carry information about the current latent state. In order to apply MDP techniques to POMDP, it is common to transform the POMD into an equivalent MDP whose states represent the agent's 's current beliefs. A belief state is a distribution b over the latent environment state. The basic idea is to transform a POMDP into an MDP by replacing latent states with the agent's beliefs about latent states. As (Russell & Norvig, 2010, Ch.17.4.1.) write in their standard textbook: "The fundamental insight...is this: the optimal action depends only on the agent s current belief state" (emphasis Russell & Norvig).
Belief Dynamics For compactness, it is customary to assume that the agent's belief state at time conditions on the available observations and actions, one writes by(8) be(s)o ac))) We adopt the standard POMDP notation for an agent's current belief state b and b' for successor belief state. Similarly, we write O, o' for an observation signal and its successor, and 2, x' for a latent state component and its successor. With these conventions, the posterior update (Equation (15)) becomes the belief update
= = x Es-Mei/PE(E/1z.0.)
(16)
Equation (16) is analogous to the standard POMDP for alomic POMDPs, with the latent state variables z replacing the latent state s. policy in belief MDP maps belief state and current observation a distribution over actions (see Figure 7b and Section 4). The Bellman equation for belief MDP is given by Equation (17).
C Bellman Equations
The Bellman equation for the observational Q-function is as follows.
((, ( + Σ ())
(17)
VT.D((o,b)) (aa
(), = B2-M(z)a) Er. pD (2
The pD terms defined by dynamic decision network shown in Section 3. This is the the standard Bellman equation for belief MDPs (Russell & Norvig, 2010, Ch.17.4.3), with in place of a latent and the epistemic state b) in place of the belief state b.
The Bellman equation for the interventional Q-function is obtained from the observational Equa- tion (17) by conditioning on the intervention do(ā) instead of the action a as follows (Wang et al., 2021; Zhang & Bareinboim, 2020).
(o, R²((o,b), do(a)) + Σ b) ((d)))
(18)
vo.D((o,b) Σ F(aKo, (o, do(a))
RD do(a)) (o(a))
Σr. PD r r| o. do(a))
25

---

Published in Transactions on Machine Learning Research (April/2025)
The what-if Bellman Equation (10) is obtained from the interventional Equation (18) by adding to the
original observation signal (0,b) the observed act a, resulting in the expanded observation signal (o, a, b,.
D Proofs
D.1 Proof of Theorem
Theorem 1. Suppose that an observation signal O is action sufficiend in dynamic probabilistic SCM D. Then for every epistemic state, observed act a, and action à:
a,, d), = do(ā))
That is, given that the direct causes of on act have been observed, the Q-value associated with an intervention à replacing the actual act equals the Q-value associated with conditioning on â, independent of the act.
We prove the result from the basic definitions rather than using derived principles like Pearl's do-calculus. The proof involves two main steps.
Show that in any causal model, the parents of an intervened variable are observed, then interven- tional, causal, and what-if probabilities are equivalent. Intuitively, if the parents A are conditioned on, then removing the links between A and its parents does remove any noncausal correlations.
Therefore under action sufficiency, causal and conditional reward and transition probabilities are equivalent. Since reward and transition probabilities determine Q-values by the Bellman equation, the equivalence of causal and conditional Q-values follows.
Lemma Let C' be probabilistic SCM and let Y, A,A be a disjoint set of random variables such that X includes all parents of A except for possibly noise variable UA of A, and none of the descendants of A. Then for any actions o, a have
pC PO
x, A a, do(A a)) PY x, = a)) -
Proof. The source variable posterior satisfies the following independence conditions:
b(u, u 2, a) b(u) = A -uu = x, A=
b(u X x, = a) b( X =x)
b(u X = x) bbubb((ux = x)) bbuul X = x)
The first independence holds because X contains all the parents of A (i.e., X, A contains the entire Markov blanket of UA). The second independence holds because X contains all the parents of A and none of its descendants, so by the Markov condition, is independent of all its non-descendants. Since u contains only source variables, it contains no descendent of A. Similarly, the third independence holds because u is independent of all its non-deseendants, and X contains no descendent of and hence no descendant of u.A.
Now consider the evaluation of the counterfactuals (Y A, a, do(A â)) and PY æ, do(A= a)). Each probability is calculated in the same submodel Sa but with different posteriors. Let G be the submodel with source posterior distribution b(u, uA u æ, A = a) and let C2 be the submodel with source posterior distribution b(u, u. X = æt.. In each submodel, A is not generated by source variables but manipulated to the value à. Let U be the set of source variables other than UA. For an assignment of values variables W w, where A W, let Uwja be the set of assignments to the source variables U such that the recursive solution procedure generates the assignment W w variable a. Together with the independence conditions above, we therefore have the following:
26

---

Published in Transactions on Machine Learning Research April/20255
pa y x)
Σ uu X = x, a)
wet
E
buu x) Σ = a)
2,
MA via
Σ
b(u = x) .1
uEU via
Σ
buu x)) Σ b(ua = x)
uEU
Uu
p y, X x)
Since the joint probabilities the same for each posterior, are the conditional counterfactual which establishos the first equality of the Lemma.
probabilities, counterrfctualpppbbbllltett.t,
The argument for the second equality is as follows: Given the asiigment pa induced by the observations æ, we can define conditional probability over actions by P(a[z) paa)pa")) = Σ /ua:fa(pe".a) b(uA), that is, summing over the set of noise variable variables that generate the observed action. The conditional and interventional distributions differ only by this term, which does not depend on the target and therefore cancels out in the conditional probability.
Our next proposition asserts the equivalence for the reward and transition models under executability.
Proposition 1. Suppose that an observation signal O is action sufficient in dynamic decision network: D. Then for every epistemic state, observed act and action à:
O A, do) = PP(ROO,,
= PP(ROO,A) A) and PO (S A). That is, the conditional, what-if interventional reward/transition probabilities are the same.
A, do(A)) = PP do(
Proof. Follows immodiately from Lemma 1.
Theorem follows from Proposition 1.
D.2 Proof of Observation
Observation 1.
Let D be a locally minimal dynammic decision network that matches a behavioral policy nz. Then ne is executable if and only if the policy observation space O is action sufficient in D.
Suppose that every parent of A is observed (i.e., Рaς S O). By Definition 1(3), the only potential descendant of A, except for successor variables, is the reward variable R. Thus the set of observed variables O contains no descendant of A. By the Markov condition, A is independent of all non-descendants given the parents of A. So given O, AA is independent all contemporaneous latent environment variables Z, which is the definition of an executable policy. The same argument applies to the successor action variable A'.
Proof. (=):
(<): Suppose that A is independent of the latent variables Z given the observed variables O. Then action- minimality requires that no latent variable is parent of A, which is the definition of action sufficiency. J
E Hindsight Counterfactuals and Policy Evaluation.
A remarkable feature of counterfactual hindsight probabilities is that they can differ from conditional proba- bilities even in action sufficient settings, such online learning. This subsection gives examples to illustrate
27

---

Published in Transactions on Machine Learning Research (April/2025)
the phenomenon, two examples for the reward model and one for policy evaluation. The general insight is that while it has long been noted in RL that past observations allow us to infer current latent states (Kaelbling et al., 1998; Hausknecht & Stone, 2015), future information also allows us to infer current latent states.
Suppose that in observed state o, an act a was followed by a reward т. We can then ask "what would the reward have been if the agent had chosen the action do(ā)" instead? The corresponding hindsight reward probability is given by counterfactual queries of the form (UR
A simple reward hindsight query in our sports example would be
PP(SCsH-1 1|CG PH SH 1, 1),
where SC = denotes the potential outcome variable "scoring a goal after intervening to shoot".
To evaluate the hindsight reward probability in the online model of Section 3.2, first we update the initial belief given the observations:
b(G 1,
=1
Informally, since the player scores only if the goalic is not healthy, we can infer from their scoring that the goalie is not healthy. Given that the player healthy and the goalie is not, the player certain to score, the hindsight reward probability is 1:
PD(SC 1|CG 1, PH 1, 1) P(SC 1|CG PH 1, GH do(SH 1))
Without hindsight, the chance of scoring only 1/2, since the goalie has 50% chance of being healthy.
A more interesting example is consider not only immediate rewards, but hindsight based on future rewards (as in hindsight credit assignment (Harutyunyan al., 2019)). Suppose that player does not take shot, and then their team scores the next time instant. Since this implies that the goalie is not healthy, infer that they would have scored if they had taken a shot earlier. Using counterfactual notation, we have
-1::r--=1CCGG- = PH = 1, OC 1) P(SC 1|CG 1, PH 1, GH 0, do(SH 1)) 1,
where before SC denotes the potential outcome variable "scoring a goal after intervening to shoot", and SC' = denotes that the obscrved outcome at the next time step that the player scored.
Hindsight Q-values can be defined by including the observed outcomes as part of the agent's observation signal. Figure 8 shows how Q-values change with hindsight. If the player is made not to shoot, they have a 50% chance of maintaining possession. they maintain possession, they will shoot at the next step according to the behavioral policy, so they are cerlain to score then because the goalie is not healthy. Hence their expected return after not shooting 1/2; in symbols
Q PH = SC 5 1,6), do = 0)) 1/2.
These examples illustrate how observing outcomes can be powerful source of information about the latent enviromment state (goalie health in our example).
F Computation of Reward Probabilities
We use the causal graphs of Figure 1 parametrized Table 5. The computations are shown in Figures
to 11.
28

---

Published in Transactions on Machine Learning Research (April/2025)
CG PH 1
==1 Q(do(SH=1)) =1 Q(do(SH=))) 1/2
CG'
SC =
do(SH=)); 50%
PH 1
V=1
b(GH=0)==1
b((GH=0)=1
do(SH=0); 50%
do:10%
P(SCSH- = 110,b) =0
P(SCsH-1 1|0,b)
v
CG
Figure 8: On-policy policy evaluation in the online setting with hindsight. The policy evaluated is the stan- dard behavioral policy (shoot if and only if close to the goal and healthy). The evaluation uses interventional reward and transition probabilities derived from the online DDN of of Section 3.2. The diagram shows the V value and Q action values for the epistemic state where the agent is observed to be close to the goal, their belief is uniform over the latent variables, and the observation signal includes the current reward (goal scored).
G Computation of Policy Values
Figure 12 is state diagram that gives the Q and V values for the marginal policy Equation (12) based (i) conditional probabilities (Figure 12a) and ii) based interventional probabilities (Figure 12b). Due to non-cxecutability, they are different for the epistemic state where CG = 1. For the conditional action values, we have CC 1,,, SH = 1) = 1/2. The epistemic state value V satisfies the equation V = 1/4 + 1/4V, so 1/3.
For the interventional action values we have CC 1,b, do = 1)) 1/4. The epistemic state value V satisfies the equation V 1/2 1/4 + 1/4V, so V 1/6.
Figure 13 shows the computation of values for the marginal policy μ, given that we observe player not taking shot when they are close to the goal. The confounded model of Figure 1b implics that the player is not healthy. Since they score only if they are healthy, it follows that their expected reward 0 regardless of what action we direct them to perform.
29

---

Published in Transactions on Machine Learning Research Appri/22255)
1/2
1/2
player health
close to goal
goalie health
player health
close to goal
goalie health
shoot
Scores goal
shoot
scores goal
(a) Observation
(b) Intervention
Figure 9: Observational and intervention probabilities in the online model of Figure la. Gray indicates observed variables whose values are specified in the query. Numbers indicate posteriors over latent variables, given the observations. Light blue indicates an intervention on a variable. Figure 9a: The observat ional scoring probability P(SC = 1|CG = 1, SH = 1, PH = 1) 1/2, the same as the probability that the goalie is healthy. Figure 9b: The query P(SC 1|CG 1. PH do SH 1)) is evaluated in the intervention model. The scoring probability remains 1/2, because both player health and shooting are observed, breaking the causal link between them has effect the scoring probability.
1
1/2
1/2
1/2
player health
close goal
goalie health
player health
close to goal
goalie health
shoot
scores goal
shoot
scores goa
(a) Observation
(b) Intervention
Figure 10: Observational and intervention probabilities in the confounded offline model of Figure 1b. The query P(SC = L|CG = 1, SH = 1) is evaluated in the observation model Figure 10a. If we see player shooting, we can infer that they are healthy. Therefore the player scores if and only if the goalie is not healthy, so the scoring probability is P(SC = 1| CG = 1, SH = 1) = 1/2. The query P(SC 1|CG = 1, do(SH = 1)) is evaluated in the intervention model Figure 10b. Without a link between player health and shooting, the probability of player and goalie health are both 1/2, which means that the scoring probability 1/4.
b(P
b1(
player health
close goal
goalie health
b'(PH)= P(SC 1|CG PH = 0 b(GH)==12
shoot
scores goal
player health
close to goal
goalle health
Abduction
shoot
spores goal
(b) Intervention and Prediction
Figure 11: Evaluating the what-if counterfactual query P(SC = L|CG = 1, SH = 0, do = 1)) for the confounded offline model of Figure 1b. Numbers indicate posterior probabilities of latent source variables given the query observations. Figure 1la, Abduction: The posterior probability of the player being healthy is 0, given that they did not. shoot. Figure 11b, Intervention and Prediction: The truncated model removes the link between Player Health and shooting and uses the posterior distribution over source variables. In the truncated model, the scoring probability is 0, given that we have inferred that the player is not healthy.
30

---

Published in Transactions on Machine Learning Research (April/202)) (025)
= 1/3
CG =1
Q[SH=1)=
b(GH=1)=1122
SH+0; 50%
Q(SH=0) =16 1/6
b(PH=1)=1/2
1) 1/4
b(GH-1))-112
do(SH-0); 50%
P(SC=1lo,b, SH=0)
b(PH=1)=1/2
P/SC=llob. do(SH=o))
SH-D; 50%
100%
do18H=0); 50%
do/SH=11;100%
PISC-1lo.b,H--t))111 1/2
PtSC-Ilo.bdoSS1111
V=0
0
V=0 C
(a) Using conditional probabilities.
(b) Using interventional probabilities.
Figure 12: Off-policy policy evaluation in the oflline setting of Figure 1b. An epistemic state comprises values for the observable variable CG and belief over the values of the latent variables GH and PH. CG - 0 is an absorbing state (once the team loses possession, the attack is over; see Section 3.2). The policy evaluated the marginal behavioral policy of Equation (12), which chooses to shoot with probability 1/2 if the player is close the goal. The evaluation uses conditional resp. interventional reward and transition probabilities derived from the DDN of Section 3.2. The diagram shows the V value and Q action values for the epistemic state where the agent is observed to be close to the goal, and their belief is uniform over the latent variables. Transitions are labelled with probabilities. ion pairs are annolated with expected rewards. Figure 12a: Policy evaluation using conditional probabilities as in Equation (17). Figure 1 12b: Policy evaluation using interventional probabilities as in Equation (18).
CG =
V=0
1) O(=)
=
do(H=))); 50%
b(GH=1)=1/2 b(PH=1)=1/2
P(SC=1lo,b, do(SH=0)) 0
(S 50%
10:00%
PISC=1lob.do(SH::)) =0
=
CG 0
Figure 13: Off-policy policy evaluation based on what-if counterfactuals in the confounded oflline model of Figure 1b. The policy evaluated is the marginal behavioral policy. The diagram shows the V value and Q action values for the epistemic state where the agent is observed to be close to the goal, their belief is uniform over the latent variables, and they do not take a shot.
31


Reviews:
[{"review_id": "4g7EHaPBL5", "reviewer": "Reviewer_qKQ2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a frustrating paper. On the one hand, the key question—when and to what extent is causal modeling relevant to RL settings?—is timely and important. On the other hand, the way it is addressed here is meandering, a bit confusing, and appears to be missing a lot of the components I would want out of an answer to that question. \n\nThe paper is billed both as a \"short\" tutorial (pg. 1), and as a statement of new results clarifying the aforementioned question, which I think is overall a weakness. There is too much tutorial material, and it is poorly organized, in the sense that it is easy for a reader not well-acquainted with this material to get lost. The authors say too many things that are not necessary for the main flow of the paper (and could be moved to SI material), which contributes to this issue. The new results are not clearly collected and stated somewhere (e.g., a technical appendix stating novel results), but scattered throughout. The core results themselves are kind of confusingly stated (e.g., Fig. 2 is misleading).\n\n**Strengths**\n\nThe authors are clearly experts in the area and competently present a large volume of technical material helpful for understanding the overall problem (when is causal reasoning necessary in RL?), as well as their attempt at a solution to it. Their conclusions appear to be technically sound. The paper will definitely be useful to people trying to understand how (and why) to integrate RL with causal reasoning. \n\n**Major weaknesses**\n\n- There is a big 'forest and trees' issue. For example, there is a lot of intro material on, e.g., CBNs (4 pages), POMDPs (5 pages), and SCMs (6 pages). As the authors go on, I don't think it's clear to readers unacquainted with this material why they're treating each topic, and what each topic contributes to the bigger picture. The goal is to formulate the standard RL POMDP setting in a way that allows causal reasoning to be applied, and in particular in a way that involves some kind of graph structure. Dynamic decision networks (DDNs), which are essentially just the idea of treating the transition model as defining a graph, is the way the authors address this issue, but this fact (core to one's ability to apply causal reasoning to RL!) I find kind of buried in the main-text discussion. I think where DDNs are first introduced (pg. 5), it is worth commenting *why* they are necessary to introduce. Point out earlier what it takes until pg. 10 to mention—that it is not a priori obvious how to port causal insights to RL settings. \n- In the authors' recent reviewer responses, they describe their paper as \"long but not dense\". I disagree, and find it both long and dense. It is not dense in the sense of being technically challenging, but dense in the sense of being kind of confusing and filled with a lot of superfluous or distracting material. There is lots and lots of repetition (in figures, in examples, in equations, ...), probably much of it unnecessary. Contributes to 'forest and trees' problem and makes it hard to understand what the important points are. Would be helpful to move lots of such material to appendices, e.g., \"The interested reader can see more on how CBNs are related to SCMs in Appendix XXX.\" (I claim that this would be *helpful*, not *hurtful*, to the learner. Learners do not know in advance which material is core and which is peripheral, so it is helpful to tell them.)\n- Tutorial material is not clearly separated from material discussing new results, which makes it harder to follow both aspects of the paper. \n- I'm not convinced that the soccer example (the main one used throughout) is ideal, since it lacks the richness of most RL problems—it's basically a bandit problem, one step and one action. Other examples in the paper (e.g., car braking) have the same issue. The paper is probably too mature to change this, but it would be nice to at least gesture at a richer toy example where the authors' conclusions are relevant.\n- The goalie diagram appears, with little modification, a total of ~ 15 times in the main text. I'm fairly confident many of these instances could be consolidated. For example, in several cases (e.g., Figs 13 and 14, Figs 5 and 6) nearly identical figures appear right next to each other and make almost identical points. These could probably be combined (ideally, not by just putting 4 nearly identical graphs in one plot, but by suitably labeling a smaller number of graphs). I think this would help, and not hurt, the naive reader's understanding of the relevant concepts.\n- The more I think about it, the more I find the flowchart depicted in Fig. 2 confusing. Whether RL is on/off-policy is independent of whether the state space is fully observable; they are orthogonal axes of variation for RL settings. The flowchart could easily be redone with \"observability, yes or no?\" as the first decision tree question. To me, this indicates maybe the sequential decision tree isn't the best way to illustrate the core finding. As far as I can tell, what the authors have actually found is that each of on-policy, online, and fully observable is individually sufficient. Why not a schematic that explicitly indicates that each of these features is on equal footing?\n- I find the discussion of related work in Sec. 8 a bit meandering. It currently reads like 'here's a bunch of stuff tangentially related to causal RL'. At least to me, given the thrust of the paper, discussing related work related to two questions is most pressing. (1) When, given previous/related work, does causal RL help in practice? (And how is this consistent with the theoretical argument presented here?) (2) How might one actually learn a causal model? Some of the material (e.g. the counterfactual regret section) seems a bit indulgent.\n\n**Minor weaknesses**\n\n- This isn't necessarily a weakness but it did confuse me. In Fig. 7b, can't (WLOG) the latest observation be included in the belief b_t?\n- The transition model depicted in Fig. 8 is confusing. I assume (given its in-text description) it is intended to describe a model where multiple shots are made. But there is no arrow that goes from the CG = 0 state to the CG = 1 state.\n- pg. 3, Go example. Kind of a nitpick, but there is indeed partial observability in the sense that the opponent's internal variables cannot be observed by the agent. Part of competitive games is modeling one's opponent, since this generally confers a strategic advantage above and beyond being able to observe the board state.\n\n\n**Formatting/typo issues**\n\n- Fig 18 is referred to 3 times on pg. 3. This could probably be reduced to once or twice.\n- Some subsection headings (e.g., 2.2, 3.2) end in periods, while others (e.g., 2.1, 3.1) don't. This doesn't matter much, but uniformity would be nice.\n- pg. 13, typo \"give several exampes\"\n- In a few places, not sure if the intent is to put equation references in parentheses. For example, pg. 13, \"the behavioral policy Equation (3) of shooting\" looks like it ought to have \"Equation (3)\" in parentheses.\n- It's a little bit weird that the two sentences which contain \"given a true causal model...\" are exactly repeated twice, once in the intro and once just before Sec. 5.2 (pg. 18). Would be better to rephrase or remove one instance.\n- pg. 20, \"RL settings With...\" heading. \"Settings\" should be capitalized and \"with\" should not be.\n- pg. 20, there should be a comma in the fictitious scenario. \"It must be ... than theirs,\" he thinks.\n- Conclusion: The citation to Pearl just says (2000), and should probably also include Pearl's name, since it is nowhere nearby.\n- pg. 31, \"Sec.7\"-> should add a space so it becomes \"Sec. 7\". There are similar instances elsewhere (e.g., \"Thm.1\", \"Fig.3b\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Recent folklore suggests that richer approaches to reinforcement learning (RL) invoke models, and *causal* models in particular. But thus far it has not been completely clear *why*, *when*, and *to what extent* causal modeling might be necessary or useful in RL settings. The authors aim to answer this question, and conclude that in RL settings that are partially observable, AND offline, AND off-policy, causal modeling is necessary to estimate value and optimal behavioral policies, since conditional probabilities are not sufficient for estimating them. \n\nThe authors also include lengthy tutorial material on relevant causality and RL concepts, including causal Bayesian networks (CBNs), partially observable Markov decision processes (POMDPs), and structural causal models (SCMs).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Much tutorial material and many parenthetical comments should probably be moved to appendices, as they seriously impede the flow of the paper.\n- It would be helpful if tutorial material were more clearly separated from results material about when interventional probabilities equal (or do not equal) conditional probabilities. For example, tutorial material on SCMs follows and precedes some results material. \n- Please include an appendix where all of the main results are clearly stated. \n- To help the reader make sense of the various Bellman equations that appear, it could be helpful to annotate them better (one- or two-word text descriptions next to each equation, e.g., state value next to V, state-action value next to Q) and to collect all of them in some technical appendix, so it is easier for readers to see their similarities and differences.\n- Please address the formatting/typo issues mentioned above.\n- Please reduce or combine some goalie diagram instances, and reduce the overall number of main text figures.\n- Please do not split the paper into two shorter ones (mentioned in reviewer responses). I think this would only make existing problems worse, and I also do not think both papers would be content-rich enough to stand on their own. The problem is not length due to too much content, but length due to low information density.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bnkq9GnOz0", "reviewer": "Reviewer_UboP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper is very thorough and presents an interesting analysis of different scenarios in which causal and regular probability estimates differ or don't in RL.\nIt serves both as a good introduction and survey of the notion of causal graphs and do-calculus in the context of RL, and presents interesting clarifying insights in its own right.\n\nI checked the math as far as possible for me, however I am not too familiar with causal inference and some detailed errors might have slipped past me. As all results seem relatively intuitive to me though, I am not too concerned that there might be major errors.\n\n### Weaknesses\n\nAn implicit assumption that is made throughout the paper, and as far as I can tell remains unstated, is that all analysis are done in an infinite sample regime in which all probabilities can be estimated correctly. In cases where online exploration and limited samples are a factor, I do not believe the authors conclusions hold. E.g. if a state has only ever been seen once, the agent cannot be sure what to credit any received reward for: its actions or idiosyncrasies of the state.\n\nRecent work [1] has argued that under exploration, any MDP turns into a POMDP, which means that most such problems (except in the unlimited sample regime) might benefit from causal structures. \n\nI think discussing this assumption and the implications of limited exploration/samples on the recommendations made could strengthen the value of the paper for RL researchers. Many might be more used to considering sample efficiency in addition to inference modelling. Note that I do not claim that the analysis in the paper is _wrong_, just that a discussion on this would greatly improve its usefulness to the community. Overall, analyzing the expressiveness of different models without accounting for sample complexity is still a very valuable contribution in its own right.\n\nAn additional question I have is whether discovering the causal structure of the environment is feasible in the strict POOO regime? If not, then this strictly necessitates pre-specifying the causal graph? If yes, then this is an important limitation for this setting in terms of learnability, that should be highlighted for the community.\n\nMinor comment: The paper is extremely long. The core results could be presented in a shorter fashion. Several tables and figures, as well as examples take up much room and could potentially be cut. For example, pages 3 and 4 take up a lot of space to present very simple well-known concepts to the reader.\n\nAs another example, the \"related work\" section takes up 5 and a half pages, which is enormous for a paper that does not aim to be a survey paper. Similarly, 7 1/2 pages of background is very thorough for a non-survey paper.\n\nThe length of the paper does detract from the quality somewhat, as I found myself getting somewhat lost while reading it. With a shorter paper that reaches its conclusion quicker, its easier for readers to follow.\n\n[1] Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability, Gosh et al., NeurIPS 21", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a thorough analysis of causal models in different reinforcement learning regimes. The authors analyze under what assumptions causally factored methods will differ from those that do not take into account the causal structure of the environment.\nThe authors conclude that specifically in the partial observable offline regime it is important to take into account additional causal structure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please discuss the assumptions on the available samples for the agent to estimate probabilities.\n\nPlease tighten the writing overall.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bOjOpveumn", "reviewer": "Reviewer_6nnX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro. (1) The discussion looks comprehensive. \n\nCon. (1) Conclusions is not clear and the paper didn't flow well, causing readability issue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper did a comprehensive discussion on searching the scenario such that conditional causal modeling is useful to reinforcement learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) Please think about how to optimize space usage. Current locations of Figures and the related text are difficult to read. \n(2) Please add more explanation after  equations .", "broader_impact_concerns": "The paper didn't provide Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We strongly suggest that the authors revise the paper based on recommendations by reviewers and try to make sure they're all accounted for. In particular, reviewer UboP still believes there is room for growth in presentation for this work to be most accessible to a new audience.
Decision: Accept (poster)