Causal Reinforcement Learning: A Survey
Authors: Zhihong Deng
Jing Jiang
Guodong Long
Chengqi Zhang
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty. Despite many remarkable achievements in recent decades, applying reinforcement learning methods in the real world remains challenging. One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world and must therefore learn from scratch through numerous trial-and-error interactions. They may also face challenges in providing explanations for their decisions and generalizing the acquired knowledge. Causality, however, offers notable advantages by formalizing knowledge in a systematic manner and harnessing invariance for effective knowledge transfer. This has led to the emergence of causal reinforcement learning, a subfield of reinforcement learning that seeks to enhance existing algorithms by incorporating causal relationships into the learning process. In this survey, we provide a comprehensive review of the literature in this domain. We begin by introducing basic concepts in causality and reinforcement learning, and then explain how causality can help address key challenges faced by traditional reinforcement learning. We categorize and systematically evaluate existing causal reinforcement learning approaches, with a focus on their ability to enhance sample efficiency, advance generalizability, facilitate knowledge transfer, mitigate spurious correlations, and promote explainability, fairness, and safety. Lastly, we outline the limitations of current research and shed light on future directions in this rapidly evolving field.

Published in Transactions on Machine Learning Research (11/2023)
Causal Reinforcement Learning: A Survey
Zhihong Deng
dew.honn.ddngg@sude.e.... uts. eduu.
Australian Artificial Intelligence Institute, University of Technology Sydney
Jing Jiang
jing.jiong edu.
Australian Artificial Intelligence Institute, University of Technology Sydney
Guodong Long
gandong.longgeenn edu.auu awn
Australian Artificial Intelligence Institute, University of Technology Sydney
Chengqi Zhang
dheengiizzvongee Buts. edu
Australian Artificial Intelligence Institute, University of Technology Sydney
Reviewed on OpenReview: https:// openreview..e.... net/ farum? id= tX9Lo
Abstract
Reinforcement learning is essential paradigm for solving sequential decision problems under uncertainty. Despite many remarkable achievements in recent decades, applying rein- forcement learning methods in the real world remains challenging. One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world and must therefore learn from scratch through numerous trial-and-error interactions. They may also face challenges in providing explanations for their decisions and gencralizing the ac- quired knowledge. Causality, however, offers notable advantages by formalizing knowledge in systematic manner and harnessing invariance for effective knowledge transfer. This has led to the emergence of causal reinforcement learning, subfield of reinforcement learn- ing that seeks to enhance existing algorithms by incorporating causal relationships into the learning process. In this survey, we provide a comprehensive review of the literature in this domain. We bogin by introducing basic concepts in causality and reinforccment learn- ing, and then explain how causality can help address key challenges faced by traditional reinforcement learning. We categorize and systematically evaluate existing causal reinforce- ment learning approaches, with focus their ability to enhance sample efficiency, advance generalizability, facilitate knowledge transfer, mitigate spurious correlations, and promote explainability, fairness, and safety. Lastly, we outline the limitations of current research and shed light on future directions in this rapidly evolving field.
1 Introduction
"All reasonings concerning matter of fact seem be founded on the relation of cause and effect. By means of that relation alone go beyond the evidence of our memory and senses."
Dovidd Hume, An Enquiry Concerning Human Understanding.
Humans possess inherent capacity to grasp the concept of causality from young age (Wellman 1992
Inagaki & Hatano 1993 Koslowski & Masnick 2002 Sobel & Sommerville 2010). This innate understanding empowers us to recognize that altering specific factors can lead to corresponding oulcomes, enabling us to actively manipulate our surroundings to accomplish desired objectives and acquire fresh insights. A deep understanding of cause and effect enables us to explain behaviors (Schult Wellman 1997), predict future outcomes (Shultz 1982), and use counterfactual reasoning to dissect past events (Harris et al.) 1996). These cognitive abilities inherently shape human thought and reasoning (Sloman 2005 Sloman & Lagnado 2015

---

Published in Transactions on Machine Learning Research (11/2023)
changing object the outcome? What about shape and color?? Are there invariants acruss troks and environments?
What the effect of certain treatments? Does receiving the treatment directly contribute the patient's recovery, or are other unmeasured factors at play?
if slowed Would I cause car accidents if had violated traffle rules regulations?
Figure 1: Illustrative examples of causality in reinforcement learning and its impact on decision making.
Pearl 2009b
Pearl & Mackenzie, 2018 forming the basis for modern society and civilization, as well as propelling advancements in science and technology (Glymour 1998).
In the pursuit of developing agents with these crucial abilities, reinforcement learning (RL) (Sutton & Barto 2018) emerges as promising path. It involves learning optimal decision decision-making policies by interacting with the environment to learn from the outcome of certain behaviors. This property makes RL naturally connected to causality, as agents can sometimes directly estimate the causal effect for policy-specified interventions. In fact, many important concerns related to causality become pronounced when we examine real-world applications of RL.
Consider the scenarios and the questions presented in Figure ! where the phrases related causality are highlighted in orange. In robotic manipulation, RI. agents need to understand the effect of altering different factors on the outcomes to avoid learning decisions sensitive to irrelevant factors. Moreover, for effective knowledge transfer, agents are expected to leverage invariance across different tasks and environments. In medical scenarios, which often involve observational studies, agents should be aware of unobserved con- founders that may remain hidden in the data generation process, such as socioeconomic status. These confounders can simultaneously affect treatments and outcomes, introducing bias into decision-making by erroneously attributing unobserved factors treatment, resulting in either overestimation or underestima- tion of the true treatment effect outcomes. Additionally, in scenarios like autonomous driving, safety concerns often lead to questions like whether the vehicle would have collided with the leading vehicle if it hadn't slowed down. Answering such queries necessitates comparing the factual world and hypothelical one in which the vehicle had not slowed down. To address this challenge, agents are expected to utilize the ability of counterfactual reasoning, which empowers them to envision and gain insights from scenarios absent from the collected data, thus improving the efficiency learning safe driving policies.
These examples underscore the potential of incorporating causality into RL, yet the journey is fraught with challenges. RL agents, lacking fundamental understanding of the world, typically rely on extensive trial and error to make rational decisions and understand causal relationships among different factors. They may also face difficulties in identifying and learning invariant mechanisms from a limited set of environments and tasks. Moreover, since traditional RL lacks a built-in capacity for modeling causality, agents may be susceptible to spurious correlations, especially in scenarios involving offline learning and partial observability. Addressing these problems necessitates careful investigation of several critical questions: how to formalize these issues, what assumptions or prior knowledge are required, how to introduce them a principled manner, and what are the consequences of making incorrect assumptions. Consequently, in recent years, researchers have been actively exploring systematic approaches to integrate causality into the realm of reinforcement learning, giving rise to an emerging field known as causal RL.
Causal RL harnesses the power of causal inference Pearl 2009b Peters el al. 2017), which offers a math- ematical framework for formalizing the data generation process and reasoning about causality (Schölkopf ct al. 2021 Kaddour et al. 2022). It an umbrella term for RL approaches that incorporate additional assumptions or knowledge about the underlying causal model inform decision-making. Modeling causality explicitly not only holds the promise of principled approach to solving complex decision-making problems
2

---

Published in Transactions on Machine Learning Research (11/2023)
but also enhances the transparency and interpretability of the decision-making process. To further claborate on the iistinctive capabilities of causal RT in addressing the challenges faced by traditional RI, we will delve into a more detailed discussion spanning sections 3 to 6 Note that causal inference, while powerful, is not a panacca; its effectiveness depends on the quality of assumptions and domain expertise applied. We will explore these nuances further section 77. to provide a balanced understanding of the role and limitations of causal RL in addressing the challenges of traditional RT.
Given the successful application of causal inference in various domains such as computer vision Lopez-Pazz et al.
Tang et al.) 2020; Wang et al., 2020b), natural language processing (Wu Feder et al. 2022), and recommender systems (Zheng et al., 2021 Zhang 2022), it is reasonable to expect that causal RL would help resolve core challenges faced by traditional RL methods and tackle new challenges in increasingly complex application scenarios.
2017:
Shen et al.
2018
ct al.
2021
Jin et al.
2021
ett al.
2021b Gao et al.
Nevertheless, a significant obstacle lies in the lack of a clear and consistent conceptualization of causal assumptions and knowledge, as they have been encoded in diverse forms across prior rescarch, tailored specific problems and objectives. The use of disparate terminologies and techniques makes it challenging to understand the essence, implications, limitations, and opportunities of causal RL, particularly for those new to the realms of cansal inference and RL. In light of this, this paper aims to provide comprehensive survey of causal RL, consolidating the diverse advancements and contributions within this field, thereby establishing meaningful connections and fostering a cohesive understanding.
Our main contributions to the field are as follows.
We present a comprehensive survey of causal RL, exploring fundamental questions such as its defi- nition, motivations, and its improvements over traditional RT. approaches. Additionally, we provide a clear and concise overview of the foundational concepts in both causality research and RL. To the best of our knowledge, this is the first comprchensive survey of causal RL in the existing RL literature
We identify the key challenges in RT. that can be effectively addressed or improved by explicitly considering causality, To facilitate a deeper understanding of the benefits of incorporating causality- aware techniques, we propose problem--riented laxonomy. Furthermore, we conduct comparative analysis of existing causal reinforcement learning approaches, examining their methodologies and limitations.
We shed light on promising research dircctions in causal RL. These include advancing theorctical analyses, establishing benchmarks, and lackling specific learning problems. As these research topics gain momentum, they will propel the application of causal RL in real-world scenarios. Hence, cstablishing a common ground for discussing these valuable ideas in this burgeoning field crucial and will foster its continuous development and success.
2
Background
To better understand causal RL, an emerging field that combines the strengths of causality research and RL, we start by introducing the fundamentals and some common concopts relovant to the two rescarch arcas.
2.1 A Brief Introduction Causality
We first. discuss how to use mathematical language to describe and study causality. In general, there are two primary frameworks that rescarchers use to formalize causality: SCMs (structural causal models) Pearl (2009a); Glymour et al. (2016) and PO (potential outcome) (Rabin 1974 Imbens & Rubin 2015). We focus on the former in this paper because it provides a graphical methodology that can help researchers abstract
We note that Schölkopf et al. (2021) and Kaddour et al. 2022 discussed causal RL alongside many other research subjects in their papers. The former mainly studied the causal representation learning problem, and the latter comprehensively investigated the field of causal machine learning. The present study, however, focuses on examining the literature on causal RL. and provides systematic review of the field.
3

---

Published in Transactions on Machine Learning Research (11/2023)
U f
SCM P:
U
Z
X
Y
f *
U+
X
Z
Y
X
Y
Z
Chain
Fork
Collider
Fruits
Scurry
(a)
(b)
Figure 2: (a) The SCM and causal graph for the scurvy example, in which consuming fruits helps to prevent scurvy by inftuencing the intake of vitamin C. (b) The three basic building blocks of causal graphs.
and better understand the data generation process. It is noteworthy that these two frameworks are logically equivalent, and most. assumptions interchangeable.
Definition 2.1 (Structural Causal Model). An SCM M is represented by tuple (V,U,F, where
V = (Vi V2, ) is set of endogenous variables that are of interest in research problem, U = UU1, U2 . model and are determined by external factors that are generally unobservable,
; is set of exogenous variables that represent the source of stochasticity in the
[f. Jm J is set of structural equations that assign values to each of the variables in V such that fi maps PA(V) UU; to V,, where PA(V)) S VVV, and U; 44
P(U) is the joint probability distribution of the exogenous variables in U.
Structural causal model provides a rigorous framework for examining how relevant features of the world interact. Each structural equation ft E F specifies the value of an endogenous variable Vú based on an exoge- nous variable U, and PV)).. These equations establish the causal links between variables and mathemati- cally characterize the underlying mechanisms of the data generation process. If all U2 E U are independent of each other, it follows that each endogenous variable V; V is independent of all its nondescendants, given its parents PV(V). In this case, we refer to the model as Markovian and PA(V)) is considered complete, it includes all the relevant immediate causes of V.. This assumption is prevalent in human discourse, probably because it allows us to omit some causes from PA(V) (which are aggregated as exogenous variables and summarized by probabilitics), forming a useful abstraction of the underlying physical processes that might be overly detailed for practical use (Pearl 2009b, Chapter 2).
(V,E), where nodes V represent endogenous variables and edges & represent causal relationships determined by the structural equations. Specifically, an edge eij E from node V, to node V, exists if the random variable VG PA(V).. In some cases, there may be causes that are omitted from PA(V)) but affect more than one variable in V. These omitted variables are referred to as unobserved confounders. In such cases, exogenous variables are not independent of each other, which leads to the loss of Markov property. we explicitly treat such variables as latent variables and represent them with nodes in the graph, the Markov property is restored (Pearl 2009b Chapter 2). Figure [a illustrates an SCM and its corresponding causal graph. This example includes three binary endogenous variables- consumption of fruits, intake of vitamin C, and occurrence of scurvy along with the relevant exogenous variables. In this example, the consumption of fruits does not directly protect the health of sailors from scurvy; instoad, it produces an indirect effect by influencing the intake of vitamin C. Therefore it is not part of the structural equation that determines sailors' health. Figure 25 introduces the three fundamental building blocks of the causal graph: chain, fork, and collider. These simple structures can be combined together to create more complex data generation processes.
Causal graph. Each SCM M is associated with a causal graph
Product decomposition involves representing a complex joint probability distribution P(V) as a product of conditional probability distributions that are easier model and analyze. By applying the chain rule, we
4

---

Published in Transactions on Machine Learning Research (11/2023)
can always decompose P(V) of n variables Vi,..,, VV, E V a product of conditional distributions:
P(V) P(V ". VV-1).
While this decomposition method is general, it does not consider the causal relationships within the data. Therefore, predecessor variables Vi, V are not necessarily the causes of V; and the conditional proba- bility of Vi is not necessarily sensitive to all predecessor variables. Assuming that the data is generated by a Markovian causal model, the causal Markov condition (Pearl, 2009b Chapter 1) helps us establish a con- nection between causation and probabilities, thereby allowing for a more parsimonious decomposition. This condition states that for every Markovian causal model with a causal graph G, the induced joint distribution P(V) is Markov relative to G. Specifically, a variable V2 E V is independent of any non-descendants given its parents PA(V)) in G. This property enables structured decomposition along causal directions, which is referred to as the causal factorization (or the discntangled factorization) (Schölkopf et al. 2021
causal Markov kernels (Ptters et al. 2017) or cousal mechanisms Schhölkppf et al. 2021). While equation [ P(V) II P(V)PA(())).
(1)
where the (conditional) probability distributions of the form P(V))A(()) are commonly referred to
is not the only method for product decomposition, it is the only one that decomposed poses P(V) as the product of
causal mechanisms. For further information regarding the product decomposition of semi-Markovian models, in which the causal graph is acyclic but the exogenous variables are not jointly independent, please refer
to Barciibboi et al. (002)).
Intervention. SCMs not only offer a rigorous mathematical framework for studying causal relationships but also facilitate the modeling of external interventions. Specifically, intervention can be encoded as alteration of some of the structural equations in M. For example, forcing the variable X = x forms a sub- To predict the outcome
model Ma, in which the set of structural equations Fg = {L : :V, & X = =x. of such an intervention, we simply employ the submodel MM to compute the new probability distribution. In addition to directly setting the variables to constant values (known as hard intervention), we can also ma- nipulate the probability distributions that govern the variables, preserving some of the original dependencies (known as soft intervention). Many research questions involve estimating the effects of interventions. For example, preventing scurvy requires identifying effective interventions (c.g., through dictary changes) that reduce the probability of getting scurvy. To distinguish from conditional probability, researchers introduced the do-operator, using P[Y]do(X = r)) to denote the intervention probability, meaning the probability distribution of the outcome variable Y when X is fixed to x. Figure 3a illustrates the difference between is noleworthy that a critical distinction between statistical
conditional and intervention probabilities. models and causal models lies in the fact that the former specifies a single probability distribution, while the latter represents a collection of distributions, one for each possible intervention (including the null in- tervention). Consequently, causal models can provide a more comprehensive understanding of the world, potentially enhancing an agent's 's robustness against certain distribution shifts (Schölkopl et al. 2021 Thams 2022
Counterfactual. Counterfactual thinking involves posing hypothetical questions, such as, "Would the scurvy patient. have stayed healthy if they had eaten enough fruit". This cognitive process allows us to rotrospectively analyze past events and reason about the potential outcomes of altering certain factors in the past. This type of thinking helps us gain insights from experiences and identify opportunities for further improvements. Since counterfactuals involve hypothetical scenarios, collecting counterfactual data is generally impossible in reality, which is the core difference between it and interventions.
In the context of SCMs, counterfactual variables are often denoted with a subscript, such as Yxm= (or Y, when there is no ambiguity) where X and Y are two sets of variables in V. This notation helps researchers differentiate counterfactuals from the original variable Y. The key difference between Y and Y, that the B 3b] provides an illustration for evaluating counterfactuals with the twin worw method (Pearl 2009b Chapter 7). The two networks represent the
latter generated by a submodel M2 in which X is set x. Figure
5

---

Published in Transactions on Machine Learning Research (11/2023)
X
Z
Y
X
U
U
Z
7
X
Z
x
Condition
Intervention
Counterfactual
P(Y)
P(Y X=
PCY | (X= (o))
PO = -
(b)
Figure 3: (a) An illustration of the difference between condition and intervention, with Y representing the health of sailors and X being discrete variable with three possible values: fruit, juice, or meat. The marginal probability distribution P(Y) studies all subgroups within the population, while the conditional probability distribution P(Y = fruit) focuses on the subgroup of sailors who have consumed fruits. contrast, the interventional probability distribution P(Y do = fruit)) examines the population in which all sailors are required consume fruits. (b) An illustration of counterfactual probabilities, which study events occurring in an imaginary world (the bottom network), e-g., considering the sick sailors who ate meat in the factual world (the upper network), would they have stayed healthy if they had consumed fruits?
factual and counterfactual (imaginary) world respectively. They are connected by the exogenous variables, sharing the same structure and variables of interest, except the counterfactual one removes arrows pointing to the variables corresponding to the hypothetical interventions. Remark that there are many different types of counterfactual quantities implied by a causal model other than the one shown in this example. In some scenarios, such as fairness analysis (Carcy & Wu 2022: Plecko & Barcinboim 2022 we may need study nested counterfactuals, e.g-; the direct and indirect ellects.
Together, correlation, intervention, and counterfactuals form the three rungs of the ladder of causation, also known the "Pearl Causal Hierarchy" (Barcinboim et al.) 2022), which naturally emerges from an SCM. This hierarchy involves increasingly refined reasoning tasks, as does the knowledge required to complete these tasks. In summary, we have provided a concise overview of several fundamental concepts in this section. For readers interested in delving deeper into the realm of causality, please refer to Appendix A and the cited references. Subsequent sections will discuss related concepts and techniques within the context of reinforcement learning.
2.2 A Brief Introduction to Reinforcement Learning
Reinforcement learning studies sequential decision problems. Mathematically, we can formalize these prob- lems Markov decision processes.
Definition 2.2 (Markov decision process). An MDP M specified by a tuple (S, P, R, 7h, where
S denotes the state space and A denotes the action space,
P: SxAxS + [0, the transition probability function that yields the probability of transitioning into the next states $t+1 after taking an action a, at the current state $t,
R S x A + is the reward function that assigns the immediate reward for taking an action a, at state 8t:
Po : S + 10 is the probability distribution that specifies the generation of the initial state, and
7 [0 denotes the discount factor that accounts for how much future events lose their value time passes.

---

Published in Transactions on Machine Learning Research (11/2023)
Markov decision processes. In definition 2.2 the decision process starts by sampling an initial state so with Mo- An agent takes responsive action using its policy æ (a function that maps state to an action) and receives a reward from the environment assigned by R. The environment evolves to a new state following P; then, the agent senses the new state and repeats interacting with the environment. The goal an RI. agent is lo search for the optimal policy x that maximizes the return (cumulative reward) Gg- In particular, at any timestep i, the return G. defined as the sum of discounted future rewards, i.c., G. = Σi= 7 Re+i- A multi-armed bandit (MAB) is special type of MDP that focuses on single-step decision-making problems. On the other hand, a partially observable Markov decision process (POMDP), generalizes the scope of MDPs by considering partial observability. In POMDP, the system still operates based on MDP, but the agent can only access a partial or incomplete description of the system state, often referred to as the observation Ot, when making decisions. For example, in a video game, a player may need to deduce the motion of a dynamic object based on the visual cues displayed on the current screen.
Value functions. The return G, evaluates how good an action sequence is. However, in stochastic en- vironments, the same action sequence can lead to diverse trajectories and consequently, different returns. Moreover, stochastic policy x outputs probability distribution over the action space. Considering these stochastic factors, the return G2 associated with a policy π becomes random variable. In order to evaluate policies under uncertainty, RL introduces the concept of value functions. There are two types of value func- tions: V"(8) (s) denotes the expected return obtained by following the policy и from state s; Q" a) denotes the expected return obtained by performing action a at state s and following the policy æ thereafter. The optimal value functions correspond to the optimal policy & are denoted by V*(s) and (s(s.)).
Bellman equations. By definition, (*(s)= [G, s] and ("(ss E, = 8, A, a). These two types of value functions be expressed in terms of one another. By expanding the return G2, rewrite value functions in recursive manner:
(*(s)= = <m((sss)
r(als) aEA
R(ssa) +7 Σ [((d)s.o)))))))
S'ES
(2)
Q" = R(s,
ΣΣ S'E
When the timestep t not specified, s and s' are often used to refer to the states of two adjacent steps. The above equations are known as the Bellman expectation equations, which establish the connection between two adjacent steps. Similarly, the Bellman optimality equations relate the optimal value functions:
(*(s) = max aEA A(s,a)++ Σ P(((ss.a)V'(())
S'ES
Q' = R(s,a) + Σ
(( max Q ab)) Y'EA
e'ES
When the environment (also referred to as the dynamic model or, simply, the model) is known, the learning problem simplifies into planning problem that can be solved using dynamic programming techniques based on the Bellman equations. However, in the realm of RL, the main focus is on unknown environments. In other words, agents do not possess complete knowledge of the transition function P(s'|s, a) and the reward function R(s, a). This characteristic brings RT closer to decision-making problems in real-world scenarios.
Categorizing reinforcement learning methods. There are several ways to categorize RL methods. One approach is based the agent's components. Policy-based methods generally focus optimizing explicitly parameterized policy to maximize the return, while value- methods use collected data to fit a value function and derive the policy implicitly from it. Actor-critic methods combine both of them, equipping an agent with both a value function and a policy. Another classification criterion is whether RL methods use an environmental model. Model-basee reinforcement learning (MBRL) methods typically employ a well-defined cnvironmental model (such as AlphaGo (Silver et al.] (017)) or construct one using the collected data. The model assists the agent in planning or generating additional training data, thereby enhancing the learning process. Furthermore, RT. can also be divided into on-policy, off-policy, and offline
7

---

Published in Transactions on Machine Learning Research (11/2023)
Ot
Dt+1
Oc+2
POMDP
St
St+1
Stt2
Initial State:
So
Use
Transition:
S faets (Se, Us(ri). for "",
Rt
Rt+1
RT+z
Reward:
R₂ SR, (S,, Ar. URA)
for t 0,
Observation:
O2 = fo, (Se, Vor)
for t
T
At
A++1
Ar+z
Policy:
A do(a~T))
for 0, "
Figure 4: An illustrative example of casting a POMDP problem into an SCM. States are marked with dashed circles to emphasize that they are latent variables in the POMDP problem, while actions are marked with hammers as they represent intervention variables controlled by policies.
approaches based on data collection. On-policy RL only utilizes data from the current policy, while off-policy RL involves data collected by other policies. Offline RL disallows data collection, restricting the agent to learn from fixed datasel.
2.3 Causal Reinforcement Learning
Before formally defining causal RL, let cast POMDP problem into an SCM M. To do this, consider the observation, state, action, and reward at each step to be endogenous variables. The observation, state transition, and reward functions are then described as deterministic functions with independent. exogenous variables, represented by a set of structural equations JF in M. This transformation is always possible using autoregressive uniformization (Buesing al. 2019), without imposing any extra constraints. allows formally discuss causality in RL, including addressing counterfactual queries that cannot be explained by non- causal methods. Figure 4 presents an illustrative example of this transformation. From this formalization, it is straightforward to derive the MDP regime by introducing an additional constraint: O, S,. In practice, states and actions may have high dimensionality, and the gramularity of the causal model can be adjusted based on our prior knowledge. While the SCM representation allows us to reason about cansality in decision- making problems and organize cansal knowledge in clear and reusable way, it does not constitute causal RL. on its own. In this paper, we define causal RL as follows.
Definition 2.3 (Causal reinforcement learning). Causal RL is an umbrella term for RL approaches that incorporate additional assumptions or prior knowledge to analyze and understand the causal mechanisms underlying actions and their consequences, enabling agents to make more informed and effective decisions.
This definition emphasizes two fundamental aspects that distinguish causal RT from non-cansal RL. 1) It emphasizes focus on causality, seeking to advance beyond superficial associations or data patterns. To meet this goal, 2) it necessitates the incorporation of additional assumptions or knowledge that accounts for the causal relationships inherent in decision-making problems.
The primary objective of RL is determine the policy π that yields the highest expected return, rather than inferring the causal effect of specific intervention. The policy that preserves the dependence of the action the state, i.e., do(a T(-Is)). Different policies result in varying trajectory distributions. As mentioned carlier, RL is close to causality because on-policy RL can directly learn the Lotal effects of actions on the outcome from interventional data. However, when we reuse observational data, as in off policoolcyy/y policy/ofilina /offline RL, the learning problem becomes more intricate, as agents may suffer from spurious correlations (Zhang et al. 2020bb Dong et al., (002)). Additionally, we may be interested in certain types of counterfactual quantities other than total effects in causal RL, as they hold the promises of improving data efficiency and performance (Bareinboim et al. 2015 Buesing al. 2019 La et al. 2020
be seen soft intervention
8

---

Published in Transactions on Machine Learning Research (11/2023)
We note that there is lack of clarity and coherence in the existing literature on causal RL, primarily because causal modeling is more of mindset than specific problem setting or solution. Previous work has explored diverse forms of causal modeling, driven by different prior knowledge and research purposes. Ideally, a perfect understanding of the data generation process would grant access to the true causal model, enabling us to answer any correlation, intervention, and even counterfactual inquiries. However, given the inherent complexity of the world, it is often impractical to access a fully specified SCM. Most of the time, we can only access data generated by this model. Unfortunately, data alone is generally insufficient to overcome the knowledge gap. The Causal Hierarchy Theorem (CHT) (Bareinboim et al.) 2022) demonstrates that the ability to address questions at one layer almost never guarantees the ability to address the questions at higher layers. Fortunately, certain forms of knowledge, such as causal graphs, may provide sufficient but attainable insights into the underlying model. They can serve as valuable surrogates, enabling us lo identify the quantities of interest as if the SCM were accessible. Note that, inferring an interventional distribution from the observational distribution and the causal graph may not always be feasible due to the presence of unobserved confounders. For more information about the identifiability issue, we refer to Bareinboim et al. (2022) Additionally, in scenarios involving multiple domains, it is often beneficial to examine invariant factors across these domains, including causal mechanisms, causal structure (causal graph), and causal representation (high-level variables that capture the causal mechanisms underlying the data). In cases where prior knowledge about these factors is lacking, we can introduce certain inductive biases, such as sparsity, independent causal mechanisms, and sparse mechanism shifts, to obtain reasonable estimates Schölkopf et al. (2021); Sontakke et al. (2021); ; Huang et al. (2022a bb.
With solid understanding of the foundational concepts and definitions, we are now well-equipped to explore the realm of causal reinforcement learning. The upcoming sections delve into four crucial challenges where causal RL demonstrates its potential: generalizability and knowledge transfer, spurious correlations, sample efficiency, and considerations beyond return, e.g., explainability, fairness, and safety.
3 Advancing Generalizability and Knowledge Transfer through Causal Reinforcement Learning
3.1 The Issue of Generalizability in Reinforcement Learning
Generalizability poses a major challenge in the deployment of RL algorithms in real-world applications. It refers to the ability of a trained policy to perform effectively in new and unseen situations (Kirk et al., 2022). The issue of training and testing in the same environment has long been a critique faced by the RI. community (Trpan] 2018). While people often expect. RT. to work reliably in different (but similar) environments or tasks, traditional RL algorithms are typically designed for solving single MDP. They can easily overfit the environment, failing to adapt to minor changes. Even in the same environment, RL algorithms may produce widely varying results with different random seeds (Zhang et al 2018a b), indicating instability and overfitting. Lanctot et al. (2017) presented an example of overfitting in multi-agent scenarios in which a well-trained RL agent struggles to adapt when the adversary slightly changes its strategy. A similar phenomenon was observed by Raghu et al. (2018). Furthermore, considering the non-stationarity and constant evolution of the real world (Hamadanian et al. 2022), there a pressing need for robust RL algorithms that can effectively handle changes. Agents should possess the capability transfer their acquired skills across varying situations rather than relearning from scratch.
3.2 How can Causality Help to Improve Generalization and Facilitate Knowledge Transfer?
Some previous studies have shown that data augmentation improves generalization (Lee et al. 2020a Wang
et al. 2020a; Yarats et al. 2021), particularly for vision-based control. This process involves generating new data by randomly shifting, mixing, or perturbing observations, which makes the learned policy more resistant to irrelevant changes. Another common practice is domain randomization. In sim-to-rcal reinforcement learning, researchers randomized the paraneters of simulators to facilitate adaptation to reality (Tobin)
et al. 2017 Peng et al. 2018). Additionally, some approaches have attempted to incorporate inductive bias
9

---

Published in Transactions on Machine Learning Research (11/2023)
T
S
S'
S
S'
S
S'
P
S
S'
A
R
A
R
A
R
N
G
A
R
(a) Generalize to noise or ir- (b) Generalize different relevant variables.
(c) Generalize to different (d) Generalize to different physical properties.
reward assignments.
goals.
Figure 5: Different types of generalization problems in reinforcement learning represented by causal graphs. In these graphs, S and S' represent states in adjacent. time steps, A represents actions, represents rewards, N represents irrelevant variables (e.g., background color), G represents goals (e.g., target position), and P represents physical properties (e.g., mass).
by designing special network structures to improve generalization performance (Kankky et al.
2017 Higgins
ct al. 2017 Zambaldi et al. 2019 Raileanu Fergus 2021)
While these works have rated empirical success, explaining why certain techniques outperform oth- ers remains challenging. This knowledge gap hinders our understanding of the underlying factors that drive successful generalization and the design of algorithms that reliably generalize in real-world scenarios. To tackle this challenge, it is essential to identify the factors that drive changes. Kirk et al. (2022) proposed using contextual MDP (CMDP) (Hallak et al. 2015) to formalize generalization problems in RI. A CMDP resembles a standard MDP but explicitly captures the variability across a set of environments or tasks, which is determined by contextual variables such goals, colors, shapes, mass, or the difficulty of game levels. From a causal perspective, these variabilities can be interpreted as different types of external interventions in the data generation process (Schölkopf et al. 2021 Thams 2022 Figure illustrates of the generalization problems corresponding to different interventions. Previous methods simulate these interventions during the training phase by augmenting original data or randomizing certain attributes, al- lowing the model to learn from various domains. By carefully scrutinizing the causal relationships behind the data, we can gain a botter understanding of the sources of generalization ability and provide a more logical explanation.
examples
More importantly, by making explicit assumptions on what changes and what remains invariant, we can derive principled methods for effective knowledge transfer and adaptation (Zhang et al. 2015 Gong et all 2016). To illustrate this point, let recall the example discussed in Figure 2a where can use X, Y, and Z to represent fruit consumption, vitamin C intake, and the occurrence of scurvy, respectively. Consider an intervention on fruit consumption (the variable X), one would have to retrain all modules in a non-causal factorization such as P(X, Y, 2) P(ZY))(X,, Z) due to the change in P(X). In contrast, with the causal factorization Y, Z) = ((X))((())((()))) only P(X) neods to be adjusted to fit the new domain. The intuition behind this example is quite straightforward: altering fruit consumption (P(X)) does not impact the vitamin C content in specific fruits (P(Z)X z)) or the likelihood of developing scurvy conditioned on the amount of vitamin C intake (P(Y)Z 2)). This property is referred to independent causal mechanisms (Schölkopf et al. ). indicating that the causal generation process compriscs stable and autonomous modules (causal mechanisms) (Pearl 2009b Chapter 2) such that changing one does not affect the others. Building on this concept, the sparse mechanism shift hypothesis (Schölkopf et al.) 2021 Perry et al. 2022) suggests that small changes the data distribution generally correspond to changes only a subset of causal mechanisms. These assumptions provide basis for designing efficient algorithms and models for knowledge transfer.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Selected methods utilizing causality to improve generalizability.
Category
Paper
Techniques
Sectings
Environments Tasks
Zhang et al.
2020a
Cansal representation learning
online
Toy
Cart-pole cdm_connrol)
Cheetah (dm control)
Trrelevant variables
Bica et al.
2021b
Causal representation learning
imitation
OpenAT Gym
MIMIC III
Wang et al.
2022
Causal dynamics learning
online'
Chemical
Manipulation (robosuite)
Saengkyongam
2022
Causal representation learning Cansal discovery
offline online
Toy
Ding et al.
2022
Manipalation (Not, accessible)
Unlock (Minigrid)
Crash (highway-env)
Sontakke et al.
2021
Cansal representation learning Intervention
offlin-to-ooulnne
Manipulation (CausalWolrd)
Lee et al. 2021
online
Manipulation (Isaac Gym)
Dynamics
Domain randomization
Zhu et al
2021
Counterfactual reasoning
ouline
Manipulation (CausalWolrd)
Guo
2022b
Mediation analysis
online
Pendulum AT Gym)
Locomotion (OpenAl Gym)
Eghbal-zade
Causal representation learning Counterfactual reasoning
online
Contextual-Gridwortt
Tisks
Pitis et al.
2022
offline
Spriteworld
Pong (Roboschool)
Manipulation (OpenAl)
Zhang & Bareinboim
2017
Causal bound
ouline"
Toy
Dynamics
Dasgupta et al. 2018
Meta learning
ouline
Toy
and Tasks
Nair al. 2019
Causal induction f
online
Light
Huang et al.
222
Causal dynamics learning
online
Cart Pole (OpenAT Gym)
Pong (OpenAI Gym)
Others
Zhu et al.
2022b
Cansal discovery
offine
Toy
Cansal dynamics learning
Inverted Pendulum (OpenAl Gym)
3.3 Causal Reinforcement Learning for Improving Generalizability
Generalization involves various settings. Zero-shot generalization entails the agent solely acquiring knowledge in training environments and then being evaluated in unseen scenarios. While this setting is appealing, it is often impractical in real-world scenarios. Alternatively, we may allow agents receive additional training in target domains, categorized Mahmoud, 2020), lifelong RL (Khetarpal et al. 2022), among others. omprehend the essential capabilities required for generalization and the expected outcomes of learning algorithms, causal RL explicitly considers the factors that govern changes in distribution. This section, therefore, categorizes existing causal RL approaches for generalization based on specific factors of change. The representative works are shown in Table D Furthermore, we also label the problem settings for these approaches, including offline, online, toiine-to-online, and imitation learning. In cases where an approach employs online training augmented by offline datasets, it is labeled as online". To be self-contained, we offer concise overview of the environments and tasks in Appendix B
transfer RL (Zhu et al., 2020), multitask RL (Vithayathil Varghese &
3.3.1 Generalize Different Environments
First, we consider how to generalize to different environments. From a causal perspective, different en- vironments share most of the causal mechanisms but differ in certain modules, resulting from different interventions in the state or observation variables. Building on the causal relationships within these vari- ables, we can categorize existing approaches into two main groups: generalization to irrelevant variables and generalization to varying dynamics.
Thhe term "toy" refers to simple, synthetically constructed datasets or simulation environments that used to experimen- tally verify findings. It is not concrete environment task. We this term consistently throughout the paper.
When the causal effect is unidentifaable, we resort the robustness of estimates against unobscrved confounding. lower bounds, which allows to
identification (partial identification), deriving its upper and
*Causal induction involves the process of extracting abstract causal variables from high-dimensionnl, low-level pixel repre- sentations, followed by recovering the underlying causal graph.
11

---

Published in Transactions on Machine Learning Research (11/2023)
To enhance the ability to generalize to irrelevant factors, RL agents must examine the causality to identify the invariance in the data generation process. Zhang et al. (2020a) investigated the problem of generaliz- ing to diverse observation spaces within the block MDP framework, such as robots equipped with different types of cameras and sensors, which is a common scenario in reality. In the block MDP framework, the observation space may be infinite, but we can uniquely determine the state (finite but unobservable) given the observation. The authors proposed using invariant prediction to learn the causal representation that generalizes to unseen observations. Similarly, Bica et al. (2021b) introduced invariant causal imitation learn- ing, which learns the imitation policy based on invariant cansal representation across multiple environments. Wang et al. (2022) studied the causal dynamics learning problem, which attempts to eliminate irrelevant variables and unnecessary dependencies so that policy learning will not be affected by these nuisance factors. Saengkyongam et al. (2022) focused on the oflline contextual bandit, problems. Their proposed approach involves iteratively assessing the invariance condition for various subsets of variables to learn an optimal invariant policy. The algorithm begins by generating a sample datasct using an initial policy and then lests the invariance of each subset across different environments. If a subset is found to be invariant, an optimal policy within that subset is learned through off- aff-policy optimization. The experimental results sug- gest that invariance is crucial for obtaining distributionally robust policies, particularly in the presence of unobserved confounders. Ding et al.] (2022) proposed an approach to address the generalization problem in goal-conditioned reinforcement learning (GCRL). Their method involves treating the causal graph as a latent variable and optimizing it using a variational likelihood maximization procedure. This method trains agents to discover causal relationships and learn a causality-aware policy that is robust against changes in irrelevant variables.
Generalizing to new dynamics is complex issue that involves different types of variations. These variations may include changes in physical properties (e-g,, gravitational acceleration), disparities between the simula- tion environment and reality, and alternations in the range of attribute values, etc. [Sontakke et al. (2021) proposed training RL agents to infer and categorize causal factors in the environment with experimental behavior learned in a self-supervised manner. These behaviors can help the agent to extract discrete causal representations from collected trajectories, which can be applicable to unseen environments, empowering the agent to effectively generalize to unseen contexts. Lee et al. (2021) proposed an approach that conducts interventions to identify relevant state variables for succossful robotic manipulation, i.c., the features that causally influence the outcome. The roboi exhibited excellent sim-to-real generalizability after training with domain randomization on the identified features. Zhuu et al.] (2021 developed an algorithm to improve the ability of agents to generalize to rarely seen or unseen object properties. This algorithm models the envi- ronmental dynamics with SCMs, allowing the agent to generate counterfactual trajectories about objects with different attribute values, which leads to improved generalizability. [uoo et al. (2022b) investigated the unsupervised dynamics generalization problem, which allows the learned model to generalize to new environments. The authors approached this challenge by leveraging the intuition that data originating from the same trajectory or similar environments should have similar properties (hidden variables encoded from trajectory segments) that lead to similar causal effects. To mcasure similarity, they employed conditional direct effects in mediation analysis. The experimental results show that the learned model performs well in new environmental dynamics.
3.3.2 Generalize to Different Tasks
Another important topic is how to generalize to different tasks. In the SCM framework, different tasks are created by altering the structural equation of the reward variable or its parent nodes on the causal graph. These tasks have the same underlying environmental dynamics, but the rewards are assigned differently.
Eghbal-zadch et al. (2021) introduced causal contextual RL, where agents aim to learn adaptive policies that can effectively adapt to new lasks defined by contextual variables. The authors proposed a contextual attention module that enables agents to incorporate disentangled features as contextual factors, leading to improved generalization compared to non-causal agents. In order make RL more effective in complex, multi-object environments, Pitis et al. (2022) suggested factorizing the stato-action space into separate local subsets. This approach allows for learning the causal dynamic model well generating counterfactual
12

---

Published in Transactions on Machine Learning Research (11/2023)
transitions in more efficient manner. By training agents counterfactual data, the proposed algorithm exhibits improved generalization to out-of-distribution tasks.
Furthermore, in reality, generalization may involve changes in both the environmental dynamics and the task. Several studies have explored this problem from a causal viewpoint. Zhang & Bareinboim (2017) investigated knowledge transfer across bandit agents in scenarios where causal effects are unidentifiable. The proposed strategy combines two steps: deriving the upper and lower bounds of causal effects using structural knowledge and then incorporating these bounds in dynamic allocation procedure to guide the search toward more promising actions in new bandit problems. The results indicated that this strategy dominates previously known algorithms and achieves faster convergence rates. Dasgupta et al. (2018) explored whether the ability to perform causal reasoning emerges from meta-learning on a simple domain with five variables. The experimental results suggested that the agents demonstrated the ability to conduct interventions and make sophisticated counterfactual predictions. These emergent abilities can effectively generalize to new causal structures. Nair al. (2019) studied the causal induction problem with visual observation. They incorporated atlention mechanisms into the agent to generate a causal graph based on visual observations and use it to make informed decisions. The experiments demonstrated that the agent effectively generalizes to new tasks and environments with unknown causal structures. More recently, Huang et al. (2022c) proposed AdaRL, a novel framework for adaptive RL that learns a latent representation with domain-shared and domain-specific components across multiple source domains. The latent representation is then used in policy optimization. This framework allows for efficient policy adaptation to new envirouments, tasks, or observations, by estimating new domain-specific parameters using small number of samples.
3.3.3 Other Generalization Problems
In oflline RL, the agent can only learn from pre-collected datasets. In this setting, agents may encounter previously unseen state-action pairs during the testing phase, leading to the distributional shift issue (Levine ct al.
2020 Most existing approaches mitigate this issue through conservative or pessimistic lcarning (Fu-
jimoto et al. 2019 Kunar et al. 2020 Yang et al., 2021b), rarely considering generalization to new states. Zhu et al. [2022b] proposed solution to generalize to unseen states. They recovered the causal structure from offline data by using causal discovery techniques, and then employ an offline MBRL algorithm to learn from the causal model. The experimental results suggested that the causal world model exhibits better generalization performance than a traditional world model, and effectively facilitates offline policy learning.
At the end of this section, it is worth noting that in the field of causal inference, there exists closely related concept known transportability. Specifically, transportability focuses on extrapolating experimen- tal findings across domains, i.e., transferring causal effects learned in experimental studies to new domains (populations/cnviion)) /environments) in which only obscrvational studics are feasible. Rescarchers have developed graphical methods and a complete algorithmic program to address this challenge (Pearl & Bareinboim 2014). Given sufficient structured knowledge (which can qualitatively determine the differences between the two populations), this algorithmic program can help us determine whether the cansal effects of the target population can be inferred from the experimental findings of the source population. Furthermore, it can elucidate what experimental and observational findings from the two populations are essential for such an inference. This setting has received limited attention in the causal RL literature, whereas many studies focus on settings that allow experiments (online data collection) to be conducted in multiple domains or use observational data to enhance online learning. Note that this problem is a very prevalent. and important aspect of scientific investigations. As researchers in both fields engage in more active knowledge exchange, we believe new and valuable research directions will emerge. For further information on transportability, please refer to Pearl & Barcinboim (2014); Bareinboim & Pearl (2016).
4 Addressing Spurious Correlations through Causal Reinforcement Learning
4.1 The Issue of Spurious Correlation in Reinforcement Learning
Making reliable decisions based solely on data inherently challenging, as correlation does not necessarily imply causation. It is important to recognize the presence of spurious correlations, which deceptive
13

---

Published in Transactions on Machine Learning Research (11/2023)
Z
Unobserved confounder
Popularity
X
?
Y
Popularity
-?-
User
preference
X
Y
Exposure
Click
Z
Collider node
Click
Does correlation between and indicate causation, or does il arise from an unobserved confounder!
Does the correlation between Xand Y indicate causation, or does it arise from collider node?
Figure 6: Causal graphs illustrating the two types of spurious correlations, with examples from real-world applications.
associations between two variables that may appear causally related but are not. These spurious correlations introduce undesired bias to the learning problem, posing significant challenge in various machine learning applications. Here are a few ralustrative examples of this phenomenon.
In recommendation systems, user behavior and preferences are often influenced by conformity, which refers to the tendency of individuals to align with larger groups social norms. Users may feel inclined to conform to popular trends recommendations. Ignoring the impact of conformity can lead to an overestimation of user's preference for certain items (Gao et al. 2022
In image classification, when dogs frequently appear alongside grass in the training set, the classifier may incorrectly label an image grass as a dog. This misclassification arises because the model relies on the background (the irrelevant factors) rather than focusing on the specific pixels that correspond to dogs (the actual causal) (Zhang et al. 2021a Wang et al. 2021c).
When determining the ranking of tweets, the of gender icons in tweets is usually not causally related to the number of likes; their statistical correlation comes from the topic, as it influences both the choice of icon and the audience. Therefore, is not appropriate to determine the ranking by gender icons Feder et al. 2022
If want to apply RL in real-world scenarios, it is important be mindful of spurious correlations, especially when the agent is working with biased data. For instance, when optimizing long-term user satisfaction in multiple-round recommendations. there is often spurious correlation between exposure and clicks in adjacent timesteps. This is because they are both influenced by item popularity. From another perspective, when we observe a click, it may depond on user preference or item popularity, which creates a spurious correlation between the two factors. both scenarios, agents may make incorrect predictions or decisions, such as only recommending popular items (a suboptimal policy for both the system and the user), and this can further cause filter bubbles. In nutshell, if the agent learns a spurious correlation between variables, it may mistakenly believe that changing one will affect the other. This misunderstanding can lead to suboptimal or even harmful behavior in real-world decision-making problems.
4.2 How can Causality Help to Address Spurious Correlations?
The non-causal approaches lack language for systematically discussing spurious correlations. From the causal perspective, spurious correlations arise when the data generation process involves unobserved con- founders (common cause) or when collider node (common effect) serves as the condition. The former leads to confounding bias, while the latter results in selection bias. See Figure X for visual interpretation of these phenomena. Causal graphs enable us to trace the source of spurious correlations by closcly scrutinizing the data generation process. To eliminale the bias induced by spurious correlations, it is necessary to make decisions regarding causality instead of statistical correlations. This is where causal reasoning comes in: It provides principled tools analyze and deal with confounding and selective bias (Pcarl 2009b Barcinboim et al., 2014; Glymour al.) 22066] Baarinnoim al. 2022). helping RL agents accurately estimate the causal effects in decision-making problems.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Selected methods utilizing causality to address spurious correlations.
Category
Paper
Techniques
Seltings imitation
Euvironments or Tasks
Zhang ct al.
2020b
Causal graph
Toy
Kumor ct a
2021
Causel graph
imitation
Toy
Swamy
2022
Instrumental variables
imitation
Lunar Lander (OpenAI Gym)
Locomotion (Py Gym)
Namkoong et al. 2020
Sensitivity analysis
offline
Toy
Bennett et al. 2121
Proxy variables
offline
Toy
Lu & Lobato
2018
Backdoor adjustment
offiine
Pendulum (OpenAI Gym)
Cart Pole (OpenAI Gym)
Confounding bias
MNIST
Zhang & Bareinboim 2019
Causel bound Sensitivity analysis
online"
Toy
Rezende ct al.
2020
Backdoor adjustment
offline
Toy
MiniPucman
3D Maze (Unity)
Zhang & Bareinhoim
2020
Cansal graph
online"
Toy
Cansal bound
Wang al
2021b
Frontdoor adjustment
online"
Backdoor adjustment
Liao et
202
Instrumental variables
offline
Guo et al.
2022a
Instrumental variables
oflline
Gasse et a..
20223
Backdoor adjustment
online"
Toy
Pace et al. 2023
Caussl graph
offline
Sepsis
HiRID
Yang et al.
2022a
Causal graph
online
Toy
Cart Pole (OpenAI Gym)
Lunar Lander (OpenAl Gym)
Zhang & Bareinhoim
2022b
Causal graph
online"
Toy
Selection bias
Bai al. (002
Inverse probability weighting
online
Manipulation (OpenAl)
Deng et al. 2201
Cansal graph
offline
DARI.
One may assume that on-policy RL is immune to spurious correlations since it directly learns causal effects of the form p(r|s, do())) from interventional data. However, it is important to note that understanding the effect of an action on the outcome alone is insufficient for comprehending the complete data generation process. The influence of different covariates on the outcome also plays a crucial rolc. For example, in personalized recommendation systems, the covariates can be divided into relevant and spurious features. If the training enviromment consistently pairs the clicked items with specific values of spurious features (e.g., clickbait in textual features), the agent may unintentionally learn policy based on these spurious features. When the feature distribution changes in the test environment, the agent may make erroneous decisions (Gao et al. 2022). This example demonstrates the prevalence of spurious correlations in real-world decision-making problems. Demystifying the causal relationships helps resolve such challenges.
4.3 Causal Reinforcement Learning for Addressing Spurious Correlations
Based on the underlying causal structure of the decision-making problem, spurious correlations manifest in two distinct types: confounding bias arising from fork structures, and selective bias arising from collider structures (as depicted by Figure Accordingly, we categorize existing methods into two groups. Addition- ally, to be self-contained, we incorporate studies on imitation learning (IL) and off-policy evaluation (OPE), given their close relevance to policy learning RL. The representative works are shown in Table
4.3.1 Addressing Confounding Bias
We start by introducing an important technique in causal inference named do-calculus Peaal 1995). It is an axiomatic system that enables us to replace probability formulas containing the do operator with ordinary conditional probabilities. The do-calculus includes three axiom schemas that provide graphical criteria
Causal graph technique refers using causal graphs describe the data generation process, and designing graphical criteria for determining properties such as identifiaillty or developing algorithms based cansal graphs.
15

---

Published in Transactions on Machine Learning Research (11/2023)
for making certain substitutions. It has been proven to be complete for identifying causal effects (Huang & Valtorta 2006 Shpitser & Pearl 2006). Derived from the do-calculus, the backdoor and frontdoor adjustment are two widely used methods for eliminating confounding bias Glymour et al. (0116) Bareinboim et al. (2022).
The koy intuition is to block spurions correlations between the treatment and outcome variables that pass through the confounders. In situations where unobserved confounding exists, it still possible to identify the causal effect of interest if observed proxy variables for the confounder are available (Miao et al.) 2018: Ghassami et al., 2023). Another popular approach for addressing unobserved confounding is to identify and employ instrumental variables (Angrist al., 1996 Baiocchi et al. 2014]. An instrumental variable, denoted as 1, must satisfy three conditions: 1) 7 is a cause of T (the treatment variable); 2) / affects Y (the outcome variable) only through T; and 3) The effect of I on Y is unconfounded. Since the instrumental variable I influences y only through T, and this effect is unconfounded, we can indirectly estimate the effect of on Y through the effect of the instrumental variable Z on Y. Furthermore, we can evaluate the robustness of the estimated causal effect against unobserved confounding by varying the strength of the confounder's effect, which is referred to as sensitivity analysis (Diaz & van der Laan 2013 Kuang et al. 2020
Zhang et al. (020b) studied single-step imitation learning problem using a combination of demonstration data and structural knowledge about the lata-generating process. They proposed a graphical criterion for determining the feasibility of imitation learning in the presence of unobserved confounders and a practical procedure for estimating valid imitating policies with confounded expert data. This approach was then extended to the sequential setting in a subsequent paper (Kumor et al., 2021). (wwmyy et al. (2022 de- signed an algorithm for imitation learning with corrupted data. They proposed to use instrumental variable regression (Stock & Trebbi 2003) to resolve the spurious correlations caused by unobserved confounding.
Several research focused on the OPE problem, which seeks to estimate the performance of policies using data generated by different policies. For example, Namkoong et al. (2220) conducted sensitivity analyses on OPE methods under unobserved confounding. They derived worst-case bounds on the performance of an evaluation policy and proposed an efficient procodure for estimating these bounds with statistical consistoncy, allowing for a reliable selection of policies. [ennetl et al.] (2021), on the other hand, proposed a new estimator for OPE in anfinite-horizon RL. The authors established the identifiability of the policy value from off-policy data by employing a latent variable model for states and actions (which can be seen proxy variables for the unobserved confounders). The authors further presented strategy for estimating the stationary distribution ratio using proxies, which is then utilized for policy evaluation.
Lu & Lobato (2018) introduced a novel approach called deconfounding RL, aiming to learn effective policies from historical data affected by unobserved factors. Their method begins by estimating a attent-variable model using observational data, which identifies latent confounders and assesses their impact on actions and rewards. Then these confounders are utilized in backdoor adjustment to address confounding bias, enabling the policy to be optimized based a deconfounding model. Experimental results demonstrate the method's superiority over traditional RL approaches when applied to observational data with confounders. Liao et al.) (2021) also focused on the offline setting. They found that RL practitioners often encounter unobscrved confounding in medical scenarios, but there are some common sources of instrumental variables, including preferences, distance to specialty care providers, and genetic variants (Baiocchi et al. 2014). Therefore, they proposed an efficient algorithm to recover the transition dynamics from observational data using instrumen- tal variables, and employ a planning algorithm, such as value iteration, to scarch for the optimal policy. Similarly, Guo al. (202aa) embraced comparable approach when lackling the POMDP problem. They addressed the estimation of the transfer kernel by framing it series of confounded regression problems that can be solved by selecting suitable instrumental variables. After constructing confidence regions for the model parameters, the final policy can be derived using a pessimistic planning method within these regions. Wang et al. (2021b), on the other hand, studied how confounded observational data can facilitate exploration during online learning. They proposed the deconfounded optimistic value iteration algorithm, which combines observational and interventional data to update the value function estimates. This algorithm effectively handles confounding bias by leveraging backdoor and frontdoor adjustment, achieving a smaller regret than the optimal onlinc-only algorithm. Gasse et al. (023) studied MBRL for POMDP problems. Specifically, They used ideas from the do-calculus framework to formulate model learning a causal infer- ence problem. They introduced a novel method to learn a latent-based causal transition model capable of
16

---

Published in Transactions on Machine Learning Research (11/2023)
St
St+1
S Si+2
X1
X2
At
At+1
T1
T2
Ze
Z
Figure 7: Causal graphs illustrating the confounded MDP (left) discussed in Wang et al. (2021b) and a DTR with two stages of treatments (right). The relationship between these two settings can be derived easily. If none of the unobserved confounders { are influenced by the states { we can aggregate them into a global confounder Z. Let Y denotes the outcome of treatments in a DTR, and let X and T represent the covariates and treatments, respectively. By decomposing the states into covariates and treatments at each step (e.g.: $2 {X1, Ti, X2)), confounded MDP reduces to DTR.
explaining both interventional and observational regimes. By utilizing the latent variable, they were able to recover the standard transition model, which, turn, facilitated the training of RL agents using simulated transitions. While the majority of existing rescarch focuses on addressing identifiable hidden confounding, such using instrumental backdoor variables, hidden confounding is possibly non-identifiable in some real-world scenarios. Pace et al. (2023) showed that even in such scenarios, careful analysis from causal perspective can still contribute to significant improvements in policy learning, Specifically, they proposed new type of uncertainty known as "delphic uncertainty", which, in contrast to the widely studied epis- temic uncertainty and aleatoric uncertainty, quantifies the variability across world models compatible with the observational data. They devised an offline RL algorithm that incorporates this uncertainty as a Bell- man penalty. Their algorithm demonstrates effectiveness in reducing non-identifiable confounding bias on two medical datasets. Yang al. (2222a) proposed the causal inference Q-network algorithm to address confounding bias arising from various types of obscrvational interferences, such Gaussian noise and obser- vation black-out. The authors began by analyzing the impact of these interferences on the decision-making process using causal graphs. They then devised a novel algorithm that incorporates the interference label to learn the relationship between interfered observations and Q-values in order to maximize rewards in the presence of observational interference. The model Icarns to infer the interference label and adjusts the policy accordingly. Experimental results demonstrate the algorithm's improved resilience and robustness against different types of interferences. Rezende al. (2020) discussed the application of partial models in RL, a model-based approach that docs not require modeling the complete (and usually high-dimenssonnl) obser- vation. They showed that the causal correctness of partial model can be influenced by behavior policies, leading to an overestimation of the rewards associated with suboptimal actions. To address this issue, the authors proposed simple yet effective solution. Since we have full control of the agent's computational graph, we can choose any node situated between the internal state and the produced action as a backdoor variable, e.g.; the intended action. By applying backdoor adjustment, we can ensure the causal correctness of partial models.
A less familiar but highly important research topic for RL researchers is dynamic treatment regimes (DTRs) (Murphy 2003). Closely related to the fields of biostatistics, epidemiology, and clinical medicine, this topic focuses on determining personalized treatment strategies, including dosing or treatment planning, It aims lo maximize the long-term clinical outcome and can be mathematically modeled as an MDP with a global confounder, as depicted in Figure| In real-world medical scenarios, global confounders such as genetic charactoristics and lifestyle often exist but may not be observed or recorded due various reasons. When applying RL learn DTRs, we must properly handle the confounders, thus, is highly relevant the topic discussed in this section.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Zhang & Bareinboim (2019) considered a setting in which causal effect is not identifiable. They proposed an online learning algorithm to solve DTRs by combining confounded observational data. Their algorithm hinges on sensitivity analyses and incorporates causal bounds to accelerate the learning process. This online learning setting has gained popularity as it allows for conducting sequential and adaptive experimentation to maximize the oulcome variable. However, a significant challenge arises when dealing with a vast space of covariates and treatments, where online learning algorithms may result in unacceptable regret. To address this challenge, Zhang & Bareinboim (2020) proposed an efficient procedure that leverages the structural knowledge encoded in the causal graph to reduce the dimensionality of the candidate policy space. By exploiting the sparsity of the graph, where certain covariates are affected by only small subset of treatments, the proposed method exponentially reduces the dimensionality of the learning problem. The experimental results consistently demonstrate that the proposed method outperforms stale-of-the-arl (SOTA) methods in terms of performance. More recently, Zhang & Bareinboim (2022b) further explored the challenge of policy space with mixed scopes, i.c., the agent has to optimize over candidate policies with varying state-action spaces. This becomes partieularly crucial in medical domains such as cancer, HIV, and depression, where finding the optimal combination of treatments is essential for achieving desired outcomes. To tackle this issue, the authors propose novel method that utilizes causal graphs to identify the maximal sets of variables that are causally related to each other. These sets are then utilized to parameterize SCMs, enabling the representation ion of different interventional distributions using a minimal set of components, thereby enhancing the efficiency of the learning process.
A notable trend that emerges from the above literature is the combination of observational and experimental data from diverse sources, commonly referred to as data fusion (Bareinboim al. 2022). This idea has been proven to be effective in addressing complex problems related to rcasoning and decision-making (Zhang & Bareeiboiim 2019 Lee et al., 2020bb Correa et al. 2021 Zhang et al.) 2022b Gasse el al. 2023). It represents an important research direction in causal inference and may have profound implications for reinforcement learning. In reinforcement learning, agents often have access to both observational and experimental data, e.g. augmenting online learning with confounded observational data collected by other policies. As a result, data fusion introduces new challenges and holds promising potential for the research community.
4.3.2 Addressing Selection Bias
Selective bias occurs when data samples fail to represent the target population. For example, selective bias arises when rescarchers seek to understand the effect of certain drug on curing a discase by investigating patients in a selected hospital. This is because those patients may differ significantly from the population regarding their residence, wealth, and social status, making them unrepresentative.
Bai ct al. (2021) | investigated the selective bias associated with using hindsight experience replay (HER) in goal-conditioned reinforcement learning (GCRL) problems. In particular, HER relabels the goal of each collected trajectory and computes new rewards, thereby allowing the agent to learn from failure experiences instances where it failed to reach the original goal but succeeded in reaching the relabeled one. How- ever, the relabeled target distribution may not accurately represent the original target distribution. This discrepancy can misguide agents trained with HER, leading them to mistakenly believe that repeating deci- sions made in the failure experience will result in high rewards. To address this issue, the authors proposed to use inverse probability weighting, a technique in causal inference, to assign appropriate weights to the rewards computed for the relabeled goals. By reweighting the samples, the agent can mitigate the selection bias induced by HER and effectively learn from a balanced mixture of successful and unsuccessful outcomes, ultimately enhancing the overall performance. [eng al. (2021) examined the offline RL problem through the lens of selective bias. In the offline setting, agents are vulnerable to the spurious correlation between uncertainty and decision-making, which can result in learning suboptimal policies. Taking a causal perspec- tive, the empirical return is the outcome of both uncertainty and actual return. Since it is infcasible to reduce uncertainty by acquiring more data in the oflline setting, an agent might mistakenly assume a causal relationship between uncertainty and actual return. As a result, it may favor policies that achieve high returns by chance (high uncertainty). To address this issuc, the authors propose quantifying uncortainty and using as penalty term in the learning process. The results show that this method outperforms various baselines that do not consider spurious correlations in the offline setting.
18

---

Published in Transactions on Machine Learning Research (11/2023)
In general, standard reinforcement learning setup allows agents to learn from the same environment which they will be tested, so issues related to sample representativennss are often not a primary concern. However, as discussed above, specific algorithriic choices or offline learning can introduce selection bias. Overall, this issue remains relatively underexplored in reinforcement learning. In the realm causal infer- ence, substantial efforts have been dedicated to addressing the challenge of selection bias. For instance, the graphical conditions proposed by Bareinboim et al. (2014) offer a method for assessing the recoverability of certain conditional probabilities and causal effects from data affected by selection bias. These efforts provide valuable tools for further investigations into this issue in reinforcement learning.
In summary, this section presents some instances of spurious correlations in reinforcement learning and the causal RL methods to solve these problems. A careful reader may notice the subtle connection between spurious correlation and generalization. For instance, we can treat unobserved confounders as contextual variables and create new domains by applying external interventions to these variables. In such cases, policies that exhibit strong generalization (robust to different contexts) tend to be less sensitive to spurious correlations, and vice versa. Therefore, the distributional robust ness framework designed for generalization may also serve as a tool to mitigate spurious correlations under certain conditions (Ding et al. 2023). However, by comparison, causal RL methods for addressing gencralization and spurious correlation may not share the same focus and techniques. The former is usually performance-orienter learned policy performs on new domains. In contrast, the latter is more focused on identifying the causal effects of interest rather than dealing with the distributional shifts associated with confounding variables. Technically, research on spurions correlations often introduces structural assumptions (c.g., causal graphs) and employs causal inference techniques like backdoor Backdoor/frontdoor /frontdoor adjustments eliminate biases. In recent years, there has been an intoresting trend (Zhagg et al., 2021a; Cui & Athey 2022 Ding et al. 2023) in exploring the intersection of the two research fields, which has the potential to offer valuable insights and techniques to researchers in both fields.
focusing on how well a
5 Enhancing Sample Efficiency through Causal Reinforcement Learning
5.1 The Issue of Sample Efficiency in Reinforcement Learning
In RL, training data typically not provided before interacting with the enviromment. Unlike supervised and unsupervised learning methods that directly learn from fixed dataset, an RL agent needs to actively gather data to optimize its policy towards achieving the highest return. An effective RI. algorithm should be able to master the optimal policy with as few experiences as possible (in other words, it need to be sample-effcciett) Current methods often require collecting millions of samples to succeed in even simple tasks, let alone more complicated environments and reward mechanisms. For example, AlphaGo Zero was trained over roughly 3 x 10" games of self-play (Silver et al. 2017;; OpenAT's Rubik's Cube robot took nearly 10 years of simulation experience (OpenAl et al. 2019). This inefficiency entails high training cost and prevents the use of RL techniques for solving world decision-making problems. Therefore, the sample efficiency issue is a core challenge in RL, necessitating the development of RL algorithms that can save time and computational resources. In this section, in addition to examining sample-effiiient algorithms that are designed to minimize regret. in online learning, we also cover a number of approaches related to knowledge transfer. These methods require only a small number of samples from the target domain to converge after pre-training on the source domains, thus indirectly enhancing the sample efficiency within the target domain.
5.2 Causal Reinforcement Learning for Addressing Sample Inefficienty
Sample efficiency has been extensively explored in RL literature (Kkaade 2003 Osband et al. 2013 Grande
ct al.] 2014 Yu 2018) and causality offers some valuable principles for designing samplo-cfficient RL al- gorithms. Accordingly, we can organize existing research into three main lines: representation learning, directed exploration, and data augmentation. The representative works are shown in Table X
19

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: Selected methods utilizing causality to optimize sample efficiency.
Category
Paper
Techniques
Settings offline to online online
Environments or Tasks
Sontakke et Loe et 2021
2021
Causal represctaation learning
Manipulation (CausalWorld)
Reprisentation Learning
Iutervention
Manipulation (Issac Gym)
Domain randomizttion
Huang ct al. 2022b
Cansal dynamics learning
ouline"
Car Racing (OpenAl Gym)
VizDuom
Wang et al.
2022
Causal dynamics learning
online"
Chemical
Manipulation (robosuite)
Directed Exploration
Seitzer et al
2021
Intervention
online
Manipulation (OpenAl)
Buesing et
2019
Counterfactual reasoning
online
Sokmbbn
Lu 2020
Counterfactual reusoning
online
Cart Pole (OpenAI Gym)
MIMIC-III
Data Augmentation
Pitis
2020
Counterfactual reasoning
offline
Spriteworld
Pong (Roboschool)
Manipulation (OpenAl)
Zhu et al. 2021
Counterfactual reasoning
online
Manipulation (CausalWorld)
5.2.1 Representation Learning for Sample Efficiency
A good representation of the enviromment can be beneficial for sample-efficient RI. By providing compact and informative representation of the enviromment, an RL agent can learn more effectively with fewer samples. This is because a good representation can help the agent identify important features of the enviromment and abstract away unnecessary details, allowing the agent to make better use of its experiences.
Motivated by the principle of independent causal mechanisms (Schölkopf al.) 2021 , Sontakke al. (2021) argue that the information in an observed trajectory is the sum of information "injected" " by different causes. Thus, when learning in multiple environments with different physical properties, if the collected trajectories are well-clusterdd, it suggests that there may be only a single causal feature that differs among these trajec- tories, such as the mass, size, or shape of an object. Based on this idea, they employ clustering performance to induce experimental behavior and use clustering results as a surrogate for causal representation. With the state augmented by these representations, the learned policies exhibit outstanding zero-shot generalization ability and require only a small number of training samples to converge in new environments.
Another way to improve sample efficiency through representations is state abstraction. Lee et al. (2021) assumed the availability of an environmental model and used causal reasoning to identify relevant con- text variables. Specifically, the proposed method conducts interventions to alter one context. variable at a time and observes the causal influence on the outcome. This approach effectively reduces the di- mensionality of the state space and simplifies the learning problem. However, in many scenarios, di- rect intervention may not be feasible. Some non- causal approaches (Jong & Stone 2005; Zhang et al.) 2022a) achieve abstraction by aggregating states that yield the same reward sequence. While these approaches help reduce the dimensionality of the state space, they still suffer from redundant dependencies and are vulnerable to spurious correlations. In contrast, causal relationships in the real world are typically sparse and stable(Schölkopf et al., 2021 Huang et al. 2022a), leading to more effective abstraction. See Figure S for a comparison between these two types of abstraction.
5
S
S
S
S
S.
S
A
Figure 8: An illustration of state transition bet adjacent time steps. (Left) No abstraction. All vari- ables are fully connected. (Middle) An irrelevant co- variate S₁ is removed but the rest are still fully con- nected. (Right) Only the causal edges are preserved.
When observations involve highddimessional and low-level data, causal representation learning helps identify the relevant high-level concepts for the given tasks. Therefore, many cansal RL approaches are motivated by the idea that exploiting the true causal structure of the problem will reduce the complexity of learning, and thus improve sample efficiency. For example, Huang et al. (2022b) introduced method to learn action-
20

---

Published in Transactions on Machine Learning Research (11/2023)
sufficient state representations from data collected by a random policy. These representations consist of minimal set of state variables that contain sufficient information for decision-making. Wang et al. (2022), on the other hand, studied task-independent state abstraction which only omits action-ireelevant variables that neither change with actions nor infiuence actions' results, identified through independence tests basod on conditional mutual information. Their approach includes variables that are potentially useful for future tasks rather than being restricted to a particular training task.
5.2.2 Directed Exploration for Sample Efficiency
While a good representation of the environment is beneficial, it is not necessarily sufficient for sample efficient 2020). To improve sample efficiency, researchers have been studying various exploration RL (Du et al.
strategies YYng al. 2022b). Some research has drawn inspiration from developmental psychology (Ryan & Deci 2000: Barto 2013) and used intrinsic motivation to motivate agents to explore unknown environments efficiently (Pathak et al. 2017 Burda et al.) (0022). This can be done by giving bonuses to exploratory behaviors that discover novel uncertain states. However, from causal perspective, it is important to note that not all regions of high uncertainty are equally important. Those regions which establish causal relationship with the task's success are more worthy of exploration.
Seitzer et al. (2021) studied the problem of directed exploration in robotic manipulation tasks, where the agent must physically interact with the target object to generate valuable data before acquiring complex manipulation skills such as object relocation. The authors proposed a method to quantify the causal influence of actions on the object and incorporated it into the exploration process to guide the agent. Experimental results demonstrate that the proposed method significantly improves the sample efficiency across various robotic manipulation tasks. On the other hand, [Sontakke et al. (2021) introduced method to learn self- supervised experiments based on the principle of independent causal mechanisms. This method utilizes the concept of Tme-Factor-at-A-TT --mee (OFAT), wherein good experimental behavior should examine one factor at a time while keeping others constant. The rationale behind this approach is that altering only one causal factor should yield less information compared to changing multiple factors simultaneously. Consequently, the learning problem of experimental behavior can be reformulated minimizing the amount of information contained in the generated data (also referred to as maximizing "causal curiosity"). Empirical results show that RL agents pre-trained with causal curiosity exhibit. improved elficiency in solving new tasks.
5.2.3 Data Augmentation for Sample Efficiency
Data augmentation is a common machine learning technique aimed at improving algorithm performance by generating additional training data. Counterfactual data augmentation uses causal model to imitate the environment and generate data that is unobserved in the real world. This is particularly useful for RT. problems because collecting large amounts of real-world data is often difficult or expensive. By simulating diverse counterfactual scenarios, RL agents can determine the effects of different actions without interacting with the cnvironment, resulting in samplc-eficicet lcarning.
causality- based approach that
The implementation of counterfactual data augmentation follows counterfactual reasoning procedure that consists of three steps (Pearl 2009a] Chapter 7), as demonstrated in Figure 0
Abduction is about using observed data to infer the values of the exogenous variables 24;
2. Action involves modifying the structural equations of the variables of interest in the SCM; and
3. Prediction uses the modified SCM to generate counterfactual data by plugging the exogenous variables back into the equations for computation.
TThis procedure intuitively elucidates the computational prucess for evaluating counterfactual statements based available evidence. Nonetheless, it is not always feasible in practice. Abduction necessitates accesss to the distribution of exugenous variables, while action and prediction require knowledge about the underlying functional relationships. In the causal inference literature, extensive research has been dedicated investigating the combination qualitative assumptions and data to identify counterfactual distributions. For more details, please refer Shpitser & Pearl (007); Correa et al. (202T]: Zhang al. (022h)
21

---

Published in Transactions on Machine Learning Research (11/2023)
FACTUAL WORLD
COUNTERFACTUAL WORLD
X
ym
2
AUGMENTATION
Different Interventions
yeo
g
ABDUCTION
ACTION
PREDICTION
X
Ux (x
= xx(x)
X
Y
== 17-1
Y = do(Y =
Y
2
Uz (x,1,2
Z = (xX,V,
Zyuy=
Intervention
Figure 9: An example counterfactual data augmentation following the counterfactual reasoning procedure: abduction, action, and prediction. The outcome of this procedure is then used to augment the training data observed in the factual world.
While traditional model-based reinforcement learning (MBRL) methods (Wang al. 2019 Luo et al. 2022 can also generate samples by fitting a probabilistic density model, they lack the capability to effectively model exogenous variables. This limitation can lead to under-fitiing when dealing with complex distributions of exogenous variables (Buesing et al. 2019). In contrast, counterfactual data augmentation explicitly incorporates exogenous variables using the SCM framework. From Bayesian perspective, traditional MBRL approaches implicitly rely on a fixed prior, such as the Gaussian distribution, for exogenous variables, whereas counterfactual data angmentation leverages additional information (evidence) from the collected data to estimate the posterior distribution of exogenous variables. Consequently, counterfactual data generation holds promise in producing high-quality training data, potentially leading to improved policy evaluation and optimization.
Buesing et al. 2009 proposed the counterfactuully-guide policy search (CF-GPS) algorithm, designed to search for the optimal policies POMDPs. They framed model-based POMDP problems using SCMs. The proposed algorithm evaluates the outcome of counterfactual actions based on real experience, thereby improving the utilization of experience data. In similar vein, [u et al. (2020) proposed sample-efficient RI. algorithm based on the SCM framework. Their objective was to address issues related to mechanism heterogeneity and data scarcity. Their approach empowers agents to evaluate the potential consequences of counterfactual actions, thereby circumventing the need for actual exploration and alleviating biases arising from limited experience. Pitis et al. (2020) presented a novel framework that leverages a locally factored dynamics model generate counterfactual transitions for RL agents. Specifically, the term "locally factored" indicates that the state-action space can be partitioned into a disjoint union of local subsets, each has its causal structure. This locally factored approach allows for exponential reduction in the sample complexity of training dynamics model and enables reliable generalization to unseen states and actions. More recently, Zhu et al. (2021) proposed a novel approach to overcome the limitations of existing MBRL methods in the context of robolic manipulation tasks. These tasks are particularly challenging due to the diversity object properties and the risk of robot damage. The proposed method uses SCMs to capture the underlying environmental dynamics and generate counterfactual episodes involving rarely seen or unsoen objects. Experimental results demonstrate superior sample efficiency, requiring fewer environment steps converge compared to existing MBRL algorithms.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: Selected methods utilizing causality for goals beyond maximizing returns.
Category
Paper
Techniques
Settings
Environments or Tasks
Foerster et al.
2018
Coutterfccual
online
StarCraft
Madumal et a.
2022
Counterfactul reasoning
online
OpenAl Gym
StarCraft
Explainabiltty
Bica et al.
2021a
Counterfactual reasoning
imitation
Toy
MIMIC-III
Mesnard al.
2021
Counterfactual reasoning
online
Toy
Key-to-Don (Not accessible)
Interleaving (Not accessible)
Tairtsis et al.
2021
Counterfactual reasoning
online
Toy
Therapy
Triantafy llou et
2022
Counterfactual reasoning
online
Goofspiel (Not accessible)
Herlau & Larsen
2022
Mediation analysis
online
Toy
DoorKey (Not accessible)
Zhang & Bareinboim 2018
Counterfactul reasoning
offine
Toy
Fairness
Mediation analysis
Huang et al. (2022c
Causal graph
online
Toy
Cansal reasoning
Balakrishnan et al.
2022
Causal graph
online
Toy
Counterfact ual reasoning
Hart & Knoll
2020
Comterfactusl reasoning
offine
BARK-MI.
Safety
Everitt al. 2021
Causal graph
online
6 Promoting Explainabilityy Fairness, and Safety with Causal Reinforcement Learning
In general, the primary objective of RT. to maximize returns. However, with the increasing integration of RL-based automated decision systems into our daily lives, becomes imperative examine the interactions between RL agents and humans, well as their potential societal implications. In this section, we explore causal RL methods that aim to address and alleviate challenges related to explainability, fairness, and safety, The representative works are shown in Table g
6.1 Explainability
6.1.1
Explainability in Reinforcement Learning
Explainability in RL refers to the ability to understand and interpret the decisions made by an RL agent. It is important to both researchers and general users. Explanations reflect the knowledge learned by the agent, facilitating in-depth understanding. They also allow researchers to participate efficiently in the design and continual optimization of an algorithm. Furthermore, explanations unveil the internal logic of the decision- making process. When agents outperform humans, we can extract valuable insights from these explanations to inform human practice within specific domain. For general users, explanations provide rationale behind each decision, thereby enhancing their comprchension of intelligent agents and instilling greater confidence in the agent's capabilities.
6.1.2 Explainable Reinforcement Learning with Causality
Explainable RL methods can be broadly categorized into two groups: post hoc and intrinsic ap- proaches (Puiutta & Veith 2020 Heuillet et al. 2021). Post hoc explanations are provided after the model execution, whereas intrinsic approaches inherently possess transparency. Post hoc explanations, such as the saliency map approach (Greydanus al.] 2018 Mott al., 2019). often rely on correlations. How- ever, as we mentioned earlier, conclusions drawn based on correlations may be unreliable and fail to answer causal questions. On the other hand, intrinsic explanations can be achieved using interpretable algorithms,
23

---

Published in Transactions on Machine Learning Research (11/2023)
such linear regression decision trees, but the limited modeling capacity of these algorithms may prove insufficient in explaining complex behaviors (Puiutta & Veith 2020).
In contrast, humans possess an innate and powerful ability to explain the connections between different events through a "mental causal model (Sloman 2005)". This cognitive ability enables us to employ causal language in our everyday interactions, using phrases such as "because," " "therefore," " and "if only." By harnessing causal relationships, we acquire natural and flexible explanations that do not rely on specific algorithms or models, thereby greatly facilitating efficient communication and collaboration. Drawing inspiration from human cognition, can integrate causality into RL to provide explanations that enable agents to articulate their decisions and comprehension of the environment and tasks using causal language. Furthermore, in cases when the agent makes mistakes, we can respond with tailored solutions guided by causal insights.
One way to generate explanations is to use the concept of counterfactuals such as exploring the minimal changes necessary to produce a different outcome. The term "counterfactual" is popular in multi-agent reinforcement learning (MARL). For example, Foorster el al. (2018) proposed a method named counterfactual multi-agent policy gradients for efficiently learning decentralized policies in cooperative multi-agent systems. More precisely, counterfactuals help resolve the challenge of multi-agent credit assignment so that agents and humans can better understand the contribution of individual behavior to the team. Some subsequent studies followed the same idea (Su et al.) 2020; Zhou et al. 2022) These approaches did not perform the
complete counterfactual reasoning procedure as shown in Figure 9 missing the critical step of abduction,
which offers opportunities for further enhancements. More recently, Triantafyllou et al. (2022) established a connection between Dec-POMMPs and SCM, enabling them to investigate the credit assignment problem in MARL using causal language. They proposed to formalize the notion of responsibility attribution based actual causality, as defined by counterfactuals, which is significant stride in developing rigorous framework that supports accountable MARL research. Mesnard et al. (2021), on the other hand, studied the temporal credit assignment problem, i.e, measuring an action's influence on future rewards. Inspired by the concept of counterfactuals from causality theory, the authors proposed conditioning value functions on future events, which separale the influence of actions on future rewards from the effects of other sources of stochasticity. This approach not only facilitates explainable credit assignment but also reduces the variance of policy gradient estimates.
Madumal et al. 2020) used theories from cognitive science to explain how humans understand the world through causal relationships and how these relationships can help us understand and explain the behavior of RL agents. They presented an approach that integrates an SCM into reinforcement learning and used the learned model to generate explanations of behavior based on counterfactual analysis. For example, when quiring why a Starcraft IT agent builds supply depots instead of barracks (a typical counterfactual query), the agent can respond by explaining that constructing supply depots is more desirable, as it helps increase the number of destroyed units and buildings. To evaluate the proposed approach, study was conducted involving 120 participants. The results demonstrate that the causality-based based explanations outperformed other explanation models in terms of understanding, explanation satisfaction, and trust. [Biaa et al. (202a)) proposed an innovative approach to gain insights into expert decision processes by integrating counterfacte reasoning into batch inverse reinforcement learning. Their method focuses on learning explanations of expert decisions by modeling their reward function based on preferences with respect to counterfactual outcomes. This framework is particularly helpful in real-world scenarios where active experimentation may not be feasible. Tsirtsis al. (2021) conducted a study on the identification of optimal counterfactual explanations | for sequential decision process. They approached this problem by formulating it a constrained search problem and devised a polynomial time algorithm based on dynamic programming to find the solution. Specifically, this problem requires the algorithm to use a causal model of the environment to search for another sequence of actions that differs from the observed sequence of actions by a specified number of actions. The study conducted by Herlau & Larsen (2022) explores the application of mediation analysis in RL. The proposed method focuses on training an RL agent to optimize natural indirect effects, which allows for identifying critical decision points. For instance, in the task of unlocking door, the agent can effectively
Remark that the term "counterfactual explanation" is commonly used in the field of explainability. It refers broad range of methods that offer explanations by analyzing the changes that could result from altering the inputs model particular way. These methods do not necessarily imply causality. For further details, please refer to Verma et al. 2022
24

---

Published in Transactions on Machine Learning Research (11/2023)
recognize the event of acquiring key. By leveraging mediation analysis, the agent can acquire concise and interpretable causal model, enhancing its overall performance and explanatory capabilities.
6.2 Fairness
6.2.1 Fairness in Reinforcement Learning
In addition to explainability, we also want agents align with human values and avoid potential harm to human society, with fairness being a key consideration. As machine learning applications continue to permeate various aspects of our daily lives, fairness is increasingly recognized as significant concern by business owners, general users, and policymakers (Carey & Wu 2022). In real-world scenarios, fairness concerns often exhibit, a dynamic nature (Gajane et al. 2022), involving multiple decision rounds. For example, resource allocation and college admissions can be modeled as MDPs (D'Amour et al. 2020 wherein actions have cumulative effects on the population, leading to dynamic changes in fairness. Ignoring this dynamic nature of system may lead to unintended consequences, c.g., exacerbating the disparity between advantaged and disadvantaged groups (Liu et al.) 2018 Creager al., 2020 D D'Amour el al. 2020).
In decision- decision-makinn problems like these, RL agents should strive to gemuinely benefit humans and promote
social good, avoiding any form of discrimination or harm towards specific individuals or groups.
6.2.2 Fair Reinforcement Learning with Causality
When considering the application of reinforcement learning to solve fairness-aware decision-making problems, it is crucial to first examine the available prior knowledge. Detailed causal modeling allows us to characterize and understand the intricate interplay between decision-making and environmental dynamics (Zhang et al. 2020c Tang al., 2023). Specifically, we can determine what obscrvable variables and latent factors are involved in specific problem and how they affect (long-term) fairness. For example, Balakrishnan et al. ( used causal graphs to study fairness principles in decision-making problems. They encoded these principles into causal, non-causal and utility components, and analyzed the relationship between different causal paths and fairness. The author then turned fairness measures into constraints to enforce certain fairness principles, thereby framing fair policy learning into a constrained optimization problem.
To ensure fairness and prevent discrimination, is sometimes necessary to consider counterfactual reason- ing (Plecko& & Bareinboim, 2022 Pearl & Mackenzic 2018 Chapter 9). For example, in Carson v. Bethlehem Steel Corporation (1996) the ruling stated that the core of discrimination issues lies in determining whether individual group would have been treated differently by altering only the sensitive attribute (c.g., sex, age, and race) while keeping other factors constant. This is apparently counterfactual statement. Zhhan decision-making & BBrrinbbii (2018) introduced the SCM framework in fair decision making problems, which provides re- searchers with a formal language to discuss fairness issues, particularly regarding counterfactual queries. Based on the causal language, the authors propose counterfactual direct effects, indirect effects, and spuri- ous effects, which correspond to different lypes of discrimination. Further, the authors derived the causal explanation formula, which quantitatively analyzes and explains the observed disparities of decisions and helps to examine the influence of discrimination within the decision-making process. Huang et al. (2022c) studied fairness in recommendation scenarios, focusing on the contextual bandits problem. They employed causal graphs to formally analyze the fairness concerns related to this problem, introducing concept known as counterfactual individual fairness. Specifically, this concept involves evaluating the expected reward an individual would have received if placed in a different sensitive group. They designed an algorithm that esti- mates the fairness discrepancy using do-calculus. The experimental results demonstrate that their proposed approach not only enhances fairness but also maintains good recommendation performance.
Various types of fairness measures have been studied extensively in fairness literature. However, as high- lighted by Kusner et al. (2017) and Zhang & Bareinboim (2018), fairness measures built upon correla- tions Zafar et al.) 2015: Wen et al. 2021). e.g., demographic parity and equal opportunity, do not explicitly differentiate the mechanismns through which sensitive altributes or other factors influence outcomes, which may increase discrimination in certain scenarios. Plecko & Bareinboim
2022 proposed a framework for
htttps:/ccaselaw. findlav. wwwdlaw.com/ussa... wwt.://ee-7bb-irrrrt///3333333 html
25

---

Published in Transactions on Machine Learning Research (11/2023)
fairness analysis through causal lens, relating the observed changes to the unobservable causal mecha- nisms. They articulated a principled framework to quantify and decompose fairness measures, as well as the interplay between these measures when examining fairness from population to an individual level. The authors also provided insightful illustrations of these concepts using a range of examples. Finally, we note that there has been a growing body of research papers investigating the intersection of causality and fair decision-making in recent years (Nabi et al., 2019 Creager et al. 2020 Zhang et al. [020cc Tang et al., 2023 Plecko & Bareinboim 2023), but how to apply causal reinforcement learning techniques solve such problems remains to be further explored.
6.3 Safety
6.3.1 Safety in Reinforcement Learning
Safety is a crucial concern in RL (Garcıa & Fernández 2013 Gu et al. 2022p. RL agents may sometimes act unexpectedlyy especially when faced with unseen situations. This issue poses a significant risk in safety- critical applications, such as healtheare or autonomous vehicles, where even a single error could have severe consequences. Additionally, RL agents may prioritize higher returns over their own safety, known as the agent safety problem (Fulton & Platzer 2018 Beard & Baheri 2022). For instance, in domains like robotic control, agents may sacrifice their lifespan for a higher mission completion rate. Addressing these safety concerns and developing robust methods to ensure safe decision-making in RL are vital for the practical deployment of RL systems in real-world applications.
6.3.2 Safe Reinforcement Learning with Causality
Safe RL problems are typically formulated as constrained MDPs (Altman 1995 1999), which extend MDPs by incorporating an additional constraint set to express various safety concerns. Existing methods primarily focus on preventing constraint violations (Achiam et al.] 2017 Chow et al. 2017). and seldom explicitly considering causality. Causal inference provides some valuable tools for studying safety, As an example, Hart & Knoll (2020) investigated the safety issue relates to autonomous driving. Researchers can conduct counterfactual policy evaluations before deploying any policy to the real world by utilizing counterfactual reasoning. The experimental results show that their method demonstrated a high success rate while signifi- cantly reducing the collision rate. On the other hand, Everitt et al. (2021) studied a critical concern known as reward tamporing, which refers to the potential for RL agents to manipulate their reward signals. This manipulation can lead to unintended consequences and undermine the elfectiveness of the learning process, thus posing a potential safety threat. In this paper, the authors presented a set of design principles aimed at developing RL agents that are robust against reward tampering, ensuring their behavior remains aligned with the intended objectives. To establish these design principles, the authors developed a causal framework supported by causal graphs, which provide a precise and intuitive understanding of the reward tampering issue. In summary, incorporating causality helps identify potential safety threats, develop preventive so- lutions, and trace the causes behind unexpected outcomes, thereby preventing RL agents from repeatedly breaching safety constraints. As a result, RL methods and systems can be employed safely and responsibly, roducing the risk of catastrophic consequenees.
We summarize the core ideas discussed spanning sections 3 to 0 by presenting Figure 10 This schematic diagram captures the various approaches to integrating causality into the reinforcement learning process, highlighting the key components and their interactions. As we conclude this section, we recognize that while significant. progress has been made in the field of causal RL, there remain important yet. underexplored avenues for future research and development. Many open problems persist within the four core challenges we have discussed. In the final section of this paper, we turn our attention to the limitations and future directions of causal RL. By examining these aspects, we aim to inspire future studies and contribute to the continued advancement of causal RI, paving the way for new breakthroughs and applications.
26

---

Published in Transactions on Machine Learning Research (11/2023)
Agent
Buffer
r0
Environment
Policy /alue
Other Envirouments
8
functions
Model
Other Sources Data
Human
A schematic diagram rating the integration of causality into the reinforcement learning process. The numbered edges represent some key components: 1) Abstraction and extraction of causal representations from raw observations; 2) Directed exploration guided by causal knowledge; 3) Fusing (pos- sibly confounded) data; 4) Incorporating causal assumptions or knowledge from humans. 5) Providing causality-based explanations; 6) Generalization and knowledge transfer; 7) Learning causal world models; 8) Counterfactual data generation; 9) Planning with world models; 10) Enhanced training of policies and value functions with causal reasoning.
Figure 10:
7 Limitations and Future Directions
7.1 Limitations
So far, we have demonstrated that causal RT methods hold great promise enhancing the decision-making capabilities of RL agents by enabling them to understand and leverage causal relationships. However, it is crucial acknowledge the limitations associated with these methods. In this part, we will outline the limitations that researchers and practitioners may encounter when employing causal RL techniques.
One of the foremost limitations lics in the roquirement for domain knowledge. Many causal RL methods heavily rely causal graphs, thus, making accurate causal assumptions is of signilicance. For example, when dealing with unobserved confounding, the use of proxy variables and instrumental variables may introduce additional risks (Martens et al. 2006 Sainani 2018; Cui et al., 2023) An inaccurate representation of the variables of interest through proxies can introduce new confounding or other forms biases. terms of instrumental variables, meeting all the rigorous conditions its definition or establishing and validating these conditions practical scenarios can be highly challenging. These limitations highlight the importance of carefully considering the underlying assumptions and the potential risks associated with the lack of knowledge.
In some real-world scenarios, the raw data is often unstructured, such as images and textual data. The causal variables involved in the data generation process may remain unknown, necessitating the development of approaches to extract causal representations from high-dimensional raw data. However, the presence of unobserved confounders may complicate the representation learning process. Learning causal model lalent space is generally infeasible without sufficient domain knowledge. Effectively extracting causal representations thus relies making certain assumptions (Schölkopf et al. 2021]. Learning causally correct model with limited prior knowledge poses a significant challenge (Rezende et al. 2020). In certain scenarios, acquiring a causal model can be more demanding than directly learning the optimal policies, which may offset the sample efficiency gains.
Morcover, scaling causal RL to complex environments presents significant challenge due to increased computational costs and model complexity. For example, the algorithm proposed by [Sontakke et al. (2021) exhibits exponential growth in complexity. To simplify the learning problem, Buesing al. (0019 assumed
27

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: The three components of causal learning.
Available information
Targets to identify
Typical questions
Causal representation learning
Ohservations
Cansal variables
What factors account for the change in position?
(representation) Causal graph
Causal discovery
Causal variables
Does mass determine the change position of an object?
Causal mechanisms learning
Causal graph
Cansal mechanisms
How mass determine the change in position of object?
access to the true transition function and reward function in their experiments, which significantly limits the practical applicability of the proposed method.
When focusing generalizability, at generalizability adopt causal representation learning or dynamics learning. These approaches generally roquire learning from multi-domain data or allowing for explicit interventions in environmental components to simulate the generation of interventional data. The quality and granularity of the learned representations closely rely on which distributional shifts, interventions or relevant signals are available (Schölkopf et al. 2021), while agents often have access only a limited number of domains in reality.
observe additional limitations. Many causal RL methods aimed
Lastly, it is important to acknowledge the limitations associated with counterfactual reasoning. Obtaining accurate and reliable counterfactual estimates often requires making strong assumptions about the underly- ing causal structure (Shpitser & Pearl 2007 Correa et al. 2021 Zhang et al.) 2022b), as counterfactuals by definition, cannot be directly observed. Some counterfactual quantities are almost never identifiable while some are identifiable under certain assumptions, such as the effect of treatment on the treated (ETT) {Pearl 2009b, Chapter 11). Furthermore, the computational complexity of counterfactual reasoning can be a bot- tleneck, especially when dealing with higl-dimensional state and action spaces. This complexity can hinder real-time decision-making in complex tasks, which remains ongoing challenge.
7.2 Causal Learning in Reinforcement Learning
3.3 and section 5.2 we explained how causal dynamics learning a class of methods closely related to MBRL can improve generalizability and sample efficiency (Wang et al. 2022 Huang et al. 2022b). These methods focus on understanding the causo-and-effect relationships between variables and the process that generates these variables. Instead of using complex, redundant connections, to model the data generation process, these methods prefer sparse, modular style. As a result, they are more efficient and stable than traditional model-based methods and allow RL agents to adapt quickly to unseen enviromments or tasks. However, we may not have perfect knowledge of the causal variables in reality. Sometimes, we must deal with high-dinensional and unstructured data like visual information. In this case, RL agents need to be able to extract causal representations from raw data (Schölkopf et al. 2021). Depending on the tasks of interest, causal representations can take various forms, ranging from abstract concepts like emotions and preferences to more concrete entities such physical objects.
In section
The complete process of learning a causal model from raw data is known as causal learning Peters et al. 2017). It is different from causal reasoning (Imbens & Rubin 2015 Glymour et al.] 2016 which only focuses on estimating specific causal effects given the causal model. Causal learning involves extracting causal representation, discovering causal relationships, and learning causal mechanisms. Table 3 briefly summarizes their characteristics. All three of these components are significant and deserve further investigation. A great deal of research has been done causal discovery (Spirtes et al. 2000 Pearl 2009b Peters et al. 2017
Vowels et al. 2022), process of recovering the causal structure of a set of variables from data, particularly concerning conditional independence tests (Spirtes et al. 2000 Sun et al., 2007 Hoyer et al. 2008 Zhang et al. 2011). Under certain assumptions, such as faithfulness, algorithuss can identify the Markov equivalence class of the underlying causal graph from observational data. is very difficult to uniquely identify a causal
graph from obsorvational data without strong assumptions about the data generation process. Therefore, some research efforts in causal inference have been done on partially identifiable causal graphs, such maximally oriented partially directed acyelic graphs (MPDAGs) (Meek 1995 Perkovic et al. 2017 Perkovic.
28

---

Published in Transactions on Machine Learning Research (11/2023)
2020j. With additional background knowledge, we can identify more causal directions, going beyond Markov equivalence classes. Nonetheless, there remains a dearth of research on effectively leveraging these partially directed graphs to address research challenges in reinforcement learning. This represents a promising area for future exploration. In addition, combining RL with causal discovery empowers an agent to actively gather interventional data from the environment to recover the underlying causal structure. This opens up an intriguing research direction that focuses on exploring methods for leveraging interventional data, or a combination of observational and interventional data, to facilitate efficient causal discovery (Addanki et al.
2020
Jaber et al. 2020 Brouillard et al. 2020 Zhu et al. 2022a
As for causal representation learning (Schölkopf et al. 2021 Wang & Jordan 2022, Shen et al. 2022). one
possible solution is to learn latent factors from figh-dimensional observations using autoencoders (Yang et al. 2021a Eghbal-zadeh et al., 2021 Tran et al.] 2022). These methods can approximatively recover causal representations and structures by virtue of carefully designed constraint terms. This idea inherently embeds an SCM into the learner, implicitly binding causal discovery and causal mochanisms learning in one solution. Additionally, in scenarios involving multiple environments or lasks, causal representations can also be derived through techniques like mining invariance (Zhang et al. 2020a Bica et al. 2021b; Saengkyongam ct al. 2022 or clustcring trajectories from diverse domains (Sontakke et al. 2021). However, determining the optimal number and granularity of causal variables remains challenging, as the optimal causal representations often depend on the specific task.
Overall, causal learning in RL is an underexplored problem and has the potential to advance the RL com- munity. Additionally, RL techniques show promise in contributing to the field of causal learning. As we delve deeper into the connections between these two fields, an exchange of insights can nurture reciprocal benefits, propelling both fields forward (Zhu et al. 2022a
7.3 Causality-aware Multitask and Meta Reinforcement Learning
Multitask reinforcement learning Parisotto et al. 2015 Teh et al.) 2017 D D'Eramo et al., 2020 Vithay-
athil Varghese & Mahmoud 2020) focuses on simultancously learning multiple tasks by sharing knowledge and leveraging synergies across tasks. This scenario commonly arises in robot manipulation, where robot needs to acquire various skills, such as reaching, grasping, and pushing. Meta-learning (Duan et al. 2016 Finn et al., 2017 Gupta et al. 2018: Xu et al. 2018), on the other hand, involves training on a task dis- tribution to gain the ability to adapt quickly to a new task. Impressive results have been achieved without explicitly considering causality, leading to the question: Is training a high-capacis model on a diverse range of tasks sufficient to gencralize to new tasks? Recent research empirically validates this hypothesis, as large pre-trained models on diverse datasets have demonstrated remarkable performance in lasks requiring few-shot learning or even zero-shot generalization ability (Brown et al.) 2020 Wei et al.) 2021).
As shown in Figure 5 different tasks essentially different interventions in the data generation pro- cess (Schölkopf et al. 2021 so it is not surprising that models trained on multiple tasks can achieve excellent generalizability they implicitly learn the causal relationships governing the data generation process of these tasks. Dasgupta et al. (2018) provide compelling evidence for this proposition. Their study demonstrates that the capability of causal inference may emerge from large-scale meta-RL. Nonetheless, testing all possible interventions and their combinations real-world scenarios is impractical. This is where causal modeling comes into play. As discussed in the previous section, causal models enable the explicit incorporation of prior knowledge, helping agents to align their understanding of the world with human cognition. Moreover, causal relationships facilitate problem abstraction (Wang et al. 2022), thereby eliminating the need of testing irrel- evant interventions. Additionally, agents that organize their knowledge using causal structures benefit from the stability of causal relationships. While non-causal agents would require finetuning the whole model for even a slightly changed task, a causality-aware agent would only need to adjust a few modules (Huang et al., 2022a), exhibiting a stronger knowledge transfer ability. These properties may also contribute lifelong (or continual) learning (Xie & Finn 2022 Khetarpal et al. 2022), allowing for fast adaptation to new tasks that arise in sequence.
29

---

Published in Transactions on Machine Learning Research (11/2023)
7.4 Human-in-the-loop Learning and Reinforcement Learning from Human Feedback
Human-in-the-lowe learning (HiLL) [Moquuera-Rey et al,, 2022) is a form of machine learning in which humans actively participate in the development cycle of machine learning models or algorithms. This can involve providing labels, preferences, or other types of feedback. When the data or task being learned is complex or requires high levels of cognition, HiLL often produces better results because humans can provide valuable insights or knowledge to the model that it may be difficult for the model to learn on its own MMosqucira- Rey ct al. 2022 Zhang & Barcinboim 2022a).
In the context of RL, HiLL typically involves incorporating human expertise into the MDP to replace the reward function that provides feedback signals. This approach allows to train RL agents with the help of human knowledge and values, alleviating the challenges associated with defining a sophisticated reward function Zhang & Bareinboim 2022a). This idea is closely related to RLHF (Reinforcement Learning from Human Feedback) (Christiano et al. 2017), a concept that has gained increasing attention recently in the training of large language models (Ziegler et al. 2020; Glaese et al. 2022 Ouyang et al. 2022), where human instructors provide rewards (or penalties) to a model to encourage (or discourage) certain behaviors. From causal perspective, humans can provide machine learning models with strong understanding of causality based on their knowledge of the world, which can help filter out bchaviors that may lead to negative outcomes. However, it is important lo nole that humans and machines may have different observalions or perceptions of the world, and uon-causal-aware RL agents may be influenced by confounding variables (Gasse et al.) 2023). In addition, we often need to consider the issue of limited budgets, as our goal is to provide meaningful leedback to RL agents at the lowest possible cost. Finally, in addition to scalar feedback, we may also provide more informative feedback to agents in the form of counterfactuals (Karalus & Lindner 2022).
7.5 Theoretical Advances in Causal Reinforcement Learning
Theoretical research in causal RL has mainly concentrated the MAB problem. Lattimore et al. (0016) first introduced the causal bandit problem, where interventions are treated as arms, and their impact on the reward and other observable covariates is associated with a known causal graph. Unlike uontextual bandits, in this setting, the observations occur after each intervention is made. Using the structural knowledge from the causal model, the agent can efficiently identify good interventions, achieving strictly better regret than algorithms that lack causal information. [Sen et al. (2017) further considered incorporating prior knowledge about the strength of interventions. Since different arms (interventions) share the same causal graph, samples from one arm can inform us about the expected value of other arms. The authors demonstrated significant theoretical and practical improvements by leveraging such information leakage.
However, these works only focused on intervening in a single node on the causal graph, where causal effects propagate only to the first-order neighbors of the intervened node. Yabe et al. 20188) generalize this sctting by studying an arbitrary set of interventions, allowing for causal effects to propagate throughout, the causal graph. [ee & Bareinboim (2018) also investigated the scenario of pulling arms in a combinatorial manner. They showed that considering all interventable variables one arm may lead to suboptimal policy, and the structural information can be used to identify the minimal intervention set that is worth intervening in. La et al. (2021) improved the naïve bounds derived in Lattimore et al. (2016), devising algorithms for causal bandit problems with sublinear cumulative regret bounds. Nair et al. (2021) study the causal bandit problem with budget constraints. In this setting, there is a trade-off between the more economical observational arm (i.e., no interventions on the causal graph) and the high-cost interventional arms. While the observational arm is less rewarding, offers valuable information for studying other arms at a significantly lower cost. More recently, Kroon et al.) (202)) introduced the concept of "separating set" from causal discovery to causal bandit problems, which renders target variable independent of a context variable when conditioned upon. By utilizing this separating set, the authors developed bandit algorithm that does not rely on the assumption of prior causal knowledge, On the other hand, Bilodeau et al.) (2022) studied the adaptivity issue concerning the d-separator, i.e., whether an algorithm can perform nearly as well as an algorithm with oracle knowledge of the presence or absence of a d-separator.
While causal bandit problems have yielded fruitful results, there have been notably fewer studies dedicated to MDP problems. As discussed in Section 4.3 some studies focused on DTRs (Zhang & Bareinboim,
30

---

Published in Transactions on Machine Learning Research (11/2023)
2019 2020). which can be modeled an MDP with global confounder variable. Additionally, there have been some investigations into confounded MDP (Zhang & Bareinboim 2016 Wang et al. 2021b), which extend the concept of DTR by admitting the presence of unobserved confounders at each time step. In the theory of causal RL, aside from understanding how causal information, especially causal graphs, enhance the regret bounds, the identifiability of causal eflects (Zhang et al. 2020b; Lu et al.] 2022) and structure (Huang et al.) 2022a) are also of great interest. While existing rescarch provides valuable insights into the theoretical foundations of causal RL, further theoretical advancements are necessary. In addition to helping comprehend the reasons behind the success of existing causal RL methods, we also hope that future theoretical advances will pave the way for designing novel and effective approaches that are both interpretable and robust for real-world applications.
7.6 Benchmarking Causal Reinforcement Learning
In RL, we are typically interested in the efficiency and convergence of the algorithm. Atari 2600 Games and Mujoco locomotion tasks (Brockman et al., 2016) are commonly used as benchmarks for discrete and continuous control problems. There are also experimental environments that evaluate the generalizability and robustness of RL, such as Procgon (Cobbe et al. 2020). Some bonchmarks focus on multitask learning, meta-learning, and curriculum learning for reinforcement. learning, such as RLBench (James et al. 2019), Meta-World
Ahmed World (Yu et al., 2021), Alchemy (Wang et al., 2021a), and Causal-World Ahmed et al. (2022). Among them, Causal-World offers wide range of robotic manipulation tasks that share a common attribute set and structure. In these tasks, the robot is required to construct goal shape using the provided building blocks. One notable feature of this benchmark is its provision of interfaces that allow manual modification of object attributes such as size, mass, and color. This enables researchers to intervene in these attributes and generate a series of tasks with consistent cansal structures.
Since causal RL is not limited to a particular type of problem, evaluation metrics may vary depending on the specific mission. While existing experimental environments have provided good benchmarks for evaluating algorithms with various metries, the underlying data generation processes in these environments often remain opaque, concealed within game simulators physics engines. This lack of transparency makes it difficult for rescarchers to fully understand the causal mechanisms behind the problems they are attempting to solve, hindering the development of the field. Recently, Ke el al. (2021) proposed a new set of environments focusing on causal discovery in visual-based RL, allowing researchers to specify the causal graph and customize its complexity. Nevertheloss, as we demonstrated in sections to 6 a large portion of the existing rescarch still relies on toy environments to evaluate the effectiveness of algorithms. Furthermore, in addition to the previously mentioned properties, a good benehmark should consider the multiple factors comprehensively, discussed in section G Thus, developing comprehensive benchmark for assessing the performance of causal RL methods remains an ongoing challenge.
7.7 Real-world Causal Reinforcement Learning
Finally, it is worth noting that the practical implementation of causal RL in real-world applications remains limited. To make it more widely applicable, we must carefully examine the challenges posed by reality. Dulac-Arnold et al. 2000 2021) identified nine critical challenges that are holding back the use of RL in the real world: limited samples; unknown and large delays; high-dimensional input; safety constraints; partial observability; multi-objective or unspecified reward functions; low latencies; offline learning; and the need for explainability.
We have discussed some of these issues throughout this paper, many of which are related to ignorance of causality. For instance, the challenge of learning from limited samples corresponds to the sample efficiency issue discussed in section 5.2) Learning from high-dimensional inputs and multiple reward functions relates to the generalization problem outlined in section 3.3 Offline learning raises concerns about spurious correlations
(section 4.3 and security and explainability are covered in section G
Although cansal models offer promising solutions to these real-world challenges, current experimental envi- ronments often fall short of meeting the research needs. As discussed in section 7.6 popular benchmarks are often treated as black boxes, and researchers have limited access to and understanding of the causal
31

---

Published in Transactions on Machine Learning Research (11/2023)
mechanisms by which these black boxes generate data. This lack of transparency significantly hinders the development of this research field. In order to facilitate the widespread adoption and application of causal RL, it is crucial to address these limitations and cultivate a culture of causal thinking.
8 Conclusion
In summary, Causal RL holds promise for tackling complex decision-making problems under uncertainty. It represents an understudied yet significant research direction. By explicitly integrating assumptions or knowledge about the causal relationship underlying the data generation process, causal RL algorithms can learn optimal policies more efficiently and make more informed decisions. In this survey, we aimed to clarify the terminologies and concepts related to causal RL and to establish connections between existing work. We proposed a problem-oriented taxonomy and systematically discussed and analyzed the latest advances in causal RL, focusing on how they address the four critical challenges facing RI.
While there is still much work to be done in this field, the results to date are encouraging. They suggest that causal RL has the potential to significantly improve the performance of RL systems in wide range of problems. Here, we summarize the key conclusions of this survey.
Causal RL is an emerging branch of RL that emphasizes understanding and utilizing causality to make better decisions (section 2.3
Causal modeling has the potential to enhance generalization (section 3.3 and sample efficiency (section 5.2), yet comes with fundamental challenges and limitations that require careful consid- eration (section 7.1) With limited causal information, RL agents may need to learn about causal representation and environmental dynamics from raw data (section 7.2
Causal effects and relationships are identifiable under certain conditions (section 2.3 and section 7.5 enabling agents to effectively utilize observational data. Furthermore, multitask learning and meta- learning may help facilitate causal learning (section 7.3 In turn, harnessing causality can enhance the ability to transfer knowledge and effectively address wide range of tasks (section 3.3
Correlation does not imply causation. Spurious correlations can lead to distorted understanding of the environment and task, resulting in suboptimal policies (section 4.3) In addition to employ- ing causal reasoning techniques, leveraging human understanding of causality may further enhance reinforcement learning (section 7.4.
In real-world applications, performance is not the only concern. Factors such explainability, fairness, and security, must also be considered (section Gh. Current benchmarks call for increased transparency and a comprchensive, multi-faccted evaluation protocol for reinforcement learning (sec- lion 7.6). which has significant implications for advancing real-world applications of causal reinforce- ment learning (section 7.7
We hope this survey will help establish connections between existing work in causal reinforcement learning, inspire further exploration and development, and provide a common ground and comprehensive resource for those looking to learn more about this exciting field.
References
Joshna Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In Interna- tional Conference on Machine Learning, pp. 22.31. PMLR, 2017.
Raghavendra Addanki, Shiva Kasiviswanathan, Andrew Megregor, and Cameron Musco. Efficiint Interven- tion Design for Causal Discovery with Latents. In Proceedings of the 37th International Conference on Machine Learning, pp. 63-73. PMLR, November 2020.
32

---

Published in Transactions on Machine Learning Research (11/2023)
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. CarsalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning. In International Conference on Learning Representations, February 2022.
Eitan Altman. Constrained Markov Decision Processes. PhD thesis, INRIA, 1995.
Eitan Altman. Constrained Markov Decision Processes: Stochastic. Modeling. Routledge, 1999.
Joshua D. Angrist, Guido W. Imbens, and Donald B. Rubin. Identification of Causal Effects Using Instru- mental Variables. Journal of the American Statistical Association, 91 (443)):4444555 June 1996. ISSN 0162-1459. doi: 10.1050/01621453 1996.1047690..
Chenjia Bai, Lingxiao Wang, Yixin Wang, Zhaoran Wang, Rui Zhao, Chenyao Bai, and Peng Liu. Addressing Hindsight. Bias in Multigoal Reinforcement Learning. IEEE Trunsaclions on Cybernelics, pp. 1 1-14, 2021. ISSN 2168-220 2275. doi: D..109//CCB..202....1.
Michael Baiocchi, Jing Cheng, and Dylan S. Small. Instrumental variable methods for causal inference. Statistics in Medicine, (311):22297-230 2340, 2014. ISSN 1097-0258, doi: 10.1002/sim.601
Sreejith Balakrishnan, Jianxin Bi, and Harold Soh. SCALES: From Fairness Principles to Constrained Decision-Mee Decision-Making, In Proceedings of the 2022 AAAI/ACM Conference AI, Ethics, and Society, AIES '22, pp. 46-55, New York, NY, USA, July 2022. Association for Computing Machinery. ISBN 978-1-4503- 9247-1. doi: 10. 10.1145/351449943353
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, (73)3))73455
July 2016. doi: 10.1073/ ppas. 1510507113.
Elias Bareinboim, Jin Tian, and Judea Pearl. Recovering from Selection Bias in Causal and Statistical Inference. Proceedings of the AAAT Conference on Artificial Intelligence, 28(1), June 2014. ISSN 2374- 3468. doi: 10.1609/ (11609/aaai.vee aaaaaaiiv288 v2811.944
Elias Barcinboim, Andrew Forney, and Judea Pearl. Bandits with Unobserved Confounders: A Causal Approach. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Elias Bareinboim, Juan D. Correa, Duligur Tbeling, and Thomas Ieard. On Pearl's hierarchy and the foun- dations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 507-556
2022.
Andrew G. Barto. Intrinsic Motivation and Reinforcement Learning. In Gianluca Baldassarre and Marco Mirolli (eds.), Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 17 47. Springer, Berlin, Heidelberg, 2013. ISBN 978-3-642-32375, doi:
Jared J. Beard and Ali Baheri. Safety Verification of Autonomous Systems: A Multi-Fidelity Reinforcement Learning Approach. arXiv preprint orXiv:2203.03// 2022.
Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Monsavi. Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1999-2007. PMLR, March 2021.
Ioana Bica, Daniel Jarrett, Alihan Hüyük, and Mihaela van der Schaar. Learning 'What-if' Explanations for Sequential Decision-Making. In International Conference on Learning Representations, March 2021a.
Ioana Bica, Daniel Jarrett, and Mihaela van der Schaar. Invariant Causal Imitation Learning for Generaliz- able Policies. In Advances in Neural Information Processing Systems, volume 34, pp. 3952-3399 -9644. Curran Associates, Inc., 2021b.
Blair Bilodeau, Linbo Wang, and Daniel M. Roy. Adaptively Exploiting d-Separators with Causal Bandits. Tn Advances in Neural Information Processing Systems, October 2022.
33

---

Published in Transactions on Machine Learning Research (11/2023)
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAT Gym, June 2016.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable Causal Discovery from Interventional Data. In Advances in Neural Information Processing Systems, volume 33, pp. 21865-21877. Curran Associates, Inc., 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Lars Buesing, Theophane Weber, Yori Zwols, Nicolas Heess, Sebastien Racaniere, Arthur Guez, and Jean- Baptiste Lespiau. Woulda, Coulda, Shoulda: Counterfactually-GGiide Policy Search. In International Conference on Learning Representations, February 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, February 2022.
Alycia N. Carey and Xintao Wu. The Causal Fairness Field Guide: Perspectives From Social and Formal Sciences. Frontiers in Big Data, 5, 2022. ISSN 2624-909X. doi: 10.3389 (0089//fdaaa.22288988883
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-61) 6120, 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martie, Shane Legg, and Dario Amodei. Deep Rein- forcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging Procedural Generation to Benchmark Reinforcement Learning, July 2020.
Juan Correa, Sanghack Lee, and Elias Bareinboim. Nested Counterfactual Tdentification from Arbitrary Surrogate Experiments. In Advances in Neural Information Processing Systems, volume 34, pp. 6856- 6867. Curran Associales, Inc., 2021.
Elliot Creager, David Madras, Toniann Pitassi, and Richard Zemel. Causal modeling for fairness in dynamical systems. In International Conference on Machine Learning, pp. 2185-2195. PMLR, 2020.
Peng Cui and Susan Athey. Stable learning establishes some common ground between causal inference and machine learning. Nature Machine Intelligence, 4(2):110-115, February 2022. ISSN 2522-5839. doi: 10. 10.1038/ 0008/442256-0220004445
Yifan Cui, Hongming Pu, Xu Shi, Wang Miao, and Eric Tchetgen Tchelgen. Semiparametric Proximal Causal Inference. Journal of the American Statistical Association, 0(0): 2023. ISSN 0162-1459. doi: (0.1080/01621455..2222 200...2222...111111..
Alexander D' Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, David Scullcy, and Yoni Halpern. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 525-534, 2020. doi: 10.1145/ 3351095.3337288..
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Podro Ortega, David Raposo, Edward Hughos, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal Reasoning from renf learning. ar Xix preprint or 1901.000 December 2018.
34

---

Published in Transactions on Machine Learning Research (11/2023)
Zhihong Deng, Zuyue Fu, Lingxiao Wang, Zhuoran Yang, Chenjia Bai, Zhaoran Wang, and Jing Jiang. SCORE: Spurious COrrelation REduction for Offline Reinforcement Learning, October 2021.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing Knowledge in Multi-Task Deep Reinforcement Learning. In International Conference Learning Representations, March 2020.
Iván Díaz and Mark J. van der Laan. Sensitivity Analysis for Causal Inference under Unmeasured Confound- ing and Measurement Error Problems. The International Journal of Biostatistics, 9(2):199 November 2013. ISSN 1557-4679. doi: 10.1515/jjb-2013-11
Wenhao Ding, Haohong Lin, Bo Li, and Ding Zhao. Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning, In Advances in Neural Information Processing Systems, October 2022.
Wenhao Ding, Laixi Shi, Yuejie Chi, and Ding Zhao. Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation, July 2023.
Simon S. Du, Sham M. Kakade, Ruosong Wang, and Lin F. Yang. Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning? In International Conference on Learning Representations, March 2020.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv preprint orXiv:1611.0277 November 2016.
Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003.1188, 2020.
Gabriel Dulac-Arnold Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis. Machine Learning, 110(9):2419 2468, 2021. doi: 10.1007/1109944--11000066
Hamid Eghbal-zadeh, Florian Henkel, and Gerhard Widmer. Learning to infer unseen contexts in causal contextual reinforcement learning. In Self-supervision for Reinforeement Learning (SSL-RL) Workshop, ICLR, pp. 11, 2021.
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, (98())) 6467, November 2021. ISSN 1573-0964. doi: 10.1007 /s1 (0007/s1122/)) 229-022100111111
Martin Faltys, Mare Zimmermann, Xinrui Lyu, Matthias Hüser, Stephanie Hyland, Gunnar Rätsch, and Tobias Merz. Hirid, high time-resolution icu dataset (version 1.1. 1). Physio. Net, 10, 2021.
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond. Transactions of the Association for Computational Linguistics, 10:1138 1158, October 2022. ISSN 2307-387X. doi: 10.1162/tacl a 10511.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 94th International Conference on Machine Learning, pp. 1126-1135. PMLR, July 2017.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirfieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAT18/IAATIS/ ////AAA//////AA./..111 EAAI'1, pp. 2974- 2982, New Orleans, Louisiana, USA, February 2018. AAAT Press. ISBN 978-1-55735-800-8.
35

---

Published in Transactions on Machine Learning Research (11/2023)
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.
PMLR, 2019.
Nathan Fulton and André Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. doi: 10.1609/ 111609/aaiiv3380 v22...321...210..
Pratik Gajane, Akrati Saxena, Maryam Tavakol, George Fletcher, and Mykola Pechenizkiy. Survey Fair Reinforcement Learning: Theory and Practice, May 2022.
Chen Gao, Yu Zheng, Wenjie Wang, Fuli Feng, Xiangnan He, and Yong Li. Causal Inference in Recommender Systems:: A Survey and Future Directions. arXic preprint arXiv:2208. 12397, August 2022.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1490, 2015.
Maxime Gasse, Damien Grasset, Guillaume Gaudron, and Pierre- Yves Oudeyer. Using Confounded Data in Latent Model-Based Reinforcement Learning. Transactions on Machine Learning Research, March 2023. ISSN 2835-8856.
AmirEmad Ghassami, Ilya Shpitser, and Eric Tchetgen Tchetgen. Partial Identification of Causal Effects Using Proxy Variables, April 2023.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillnnhaam, Jonathan Uesato, Po- Sen Huang, Ramona Comancscu, Fan Yang. Abigail See, Sumanth Dathathri, Rory Greig, Charlic Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, lason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, September 2022.
Clark Glymour. Review of The Art of Causal Conjecture. Journal of the American Statistical Association, 93(444):1513311115 1998. ISSN 0162-1459. doi: 10.2307 /267047
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019. doi:
Madelyn Glymour, Judea Pearl, and Nicholas P. Jewell. Caasal Inference in Statistics: Primer. John Wiley Sons, 2016.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Do- main Adaptation with Conditional Transferable Components. In Proceedings of The 33rd International Conference Machine Learning, pp. 2839 2848. PMLR, June 2016.
Robert Grande, Thomas Walsh, and Jonathan How. Sample Efficient Reinforcement Learning with Gaussian Processes. In Proceedings of the 31st International Conference on Machine Learning, pp. 1332-1340 PMLR, June 2014.
Samuel Greydamus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International Conference Machine Learning, pp. 1792-1801. PMLR, 2018.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A Review of Safe Reinforcement Learning: Methods, Theory and Applications. arXiv preprint arXiv:2205. 2T1::2255100390 2022.
Hongyi Guo, Qi Cai, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Provably Efficient Offline Re- inforcement Learning for Partially Observable Markov Decision Processes. In Proceedings of the 39th International Conference on Machine Learning, pp. 8016 8038. PMLRR June 2022a.
36

---

Published in Transactions on Machine Learning Research (11/2023)
Jiaxian Guo, Mingming Gong, and Dacheng Tao. A Relational Intervention Approach for Unsupervised Dy- namics Generalization in Model-Based Reinforcement Learning. In International Conference on Learning Representations, March 2022b.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- Reitforeinfrr Learning of Structured Exploration Strategies. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Assal Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprinl arXiv:1502.02255 2015.
Pouya Hamadanian, Malte Schwarzkopf, Siddartha Sen, and Mohammad Alizadeh. How Reinforcement Learning Systems Fail and What to do About It. The 2nd Workshop on Machine Learning and Systems (BuroMLSyss) 2022.
Paul L. Harris, Tim German, and Patrick Mills. Children's 's use of counterfactual thinking in causal reasoning. Cognition, 61 (133): 259, December 1996. ISSN 0010-0277. doi: 10.1016 114106///00100027779000077555550
Patrick Hart and Alois Knoll. Counterfactual Policy Evaluation for Decision- Makion in Autonomous Driving, November 2020.
Tue Herlau and Rasinus Larsen. Reinforcement Learning of Causal Variables Using Mediation Analysis. Proceedings of the A AAAT Conference on Artificial Intelligence, (6(6):6910-691) June 2022. ISSN 2374- 3468. doi: 10.1609/ 2011699/aaan www.a..3i.......... 20648.
Alexandre Heuillet, Fabien Couthouis, and Natalia Dfaz-Rodríguez. Explainability in deep reinforcement learning. Knowledgee- Based Systems, 214:106685, 2021. doi: 10. http::/kwossss....1000
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1480-1490. PMLR, 2017.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008.
Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning. In International Conference on Learning Representations, May 2022a.
Biwei Huang, Chaochao Lu, Liu Leqi, Jose Miguel Hernandez-Lobato, Clark Glymour, Bernhard Schölkopf, and Kun Zhang. In Proceedings of the 39th International Conference on Machine Learning, pp. 9260 9279. PMLR, June 2022b.
State Representation Learning for Control with Structural Constraints.
Wen Huang, Lu Zhang, and Xintao Wu. Achieving Counterfactual Fairness for Cansal Bandit. Proceedings of the AAAI Conference on Arlificial Inteligence, 36(6):6952 6959, June 2022c. ISSN 2374-3468. doi: 10.1609 10.1609/anai.va wwa...3....2006..
Yimin Huang and Marco Valtorta. Pearl's calculus of intervention complete. In Proceedings of the Twenty- Second Conference on Uncertainty in Artificial Intelligence, UAI'06, pp. 217-224, Arlington, Virginia, USA, July 2006. AUAI Press. ISBN 978-0-9749009--277.
Stephanie L. Hyland, Martin Faltys, Matthias Hüser, Xinrui Lyu, Thomas Gumbsch, Cristóbal Esteban, Christian Bock, Max Horn, Michael Moor, Bastian Rieck, Mare Zimmermann, Dean Bodenham, Karsten Borgwardt, Gunnar Rätsch, and Tobias M. Merz. Early prediction of circulatory failure the intensive care unit using machine learning. Nature Medicine, 26(3):364-37) March 2020. ISSN 1546- 1546-170X. doi: 10. 10.1038/s41580 1003///41591-0000007888
37

---

Published in Transactions on Machine Learning Research (11/2023)
Guido W. Imbens and Donald B. Rubin. Causal Inference in Stotistics, Social, and Biomedical Sciences. Cambridge University Press, 2015.
Kayoko Inagaki and Giyoo Hatano. Young Children's Understanding of the Mind-Body Distinction. Child Development, 64(5):1534-1549) 1993. ISSN 1467-8624. doi: 1011111..467-6644100088000000000
Alex Trpan. Deep Reinforcement Learning Doesn't ' Work Yet, 2018.
Amin Jaber, Murat Kocaogiu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal Discovery from Soft Interventions with Unknown Targets: Characterization and Learning. In Adoances in Neural Information Processing Systems, volume 33, pp. 9551-9561. Curran Associates, Inc., 2020.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The Robot Learning Benchmark & Learning Environment, September 2019.
Zhijing Jin, Julius von Kügglgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Mrinmaya. Sachan, and Bern- hard Schoelkopf. Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Pro- cessing, pp. 9499 9513, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1, /20///////221.////// 20000..mmlpmmaaii.....
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Nicholas K. Jong and Peter Stone. State Abstraction Discovery from Irrelevant State Variables. In IJCAI, volume 8, pp. 752-757. Citeseer, 2005.
Jean Kaddour, Aengus Lynch, Qi Liu, Matt Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. Xi preprint arXiv:2206.150 15475, 2022.
Sham Machandranath Kakade. On the Sample Complezity of Reinforcement Learning. PhD thesis, University College London, March 2003.
Ken Kansky, Tom Silver, David A. Mély, Mohamed Eldawy, Miguel Greailll,,,,, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. In International Conference on Machine Leurning, pp. 1809 1818. PMLR, 2017.
Jakob Karalus and Felix Lindner. Accelerating the Learning of TAMER with Counterfactual Explanations. arXiv preprint Xix: 01358, 2022.
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Christopher Pal, and Michael Mozer. Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning, arXiv preprint Xrx:::2107.0000 00848, pp. 13, 2021.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards Continual Reinforcement Learning: A Review and Perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, December 2022. ISSN 1076-9757. doi: 00.1613/jair.1.1307
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A Survey of Generalisation in Deep Reinforcement Learning. arXiv preprint arX:::2111.0979/, Jamuary 2022.
Barbara Koslowski and Amy Masnick. The Development of Causal Reasoning. In Usha Goswami (ed.), Blackwell Handbook of Childhood Cognitive Development, pp. 257 281. Blackwell Publishers Ltd, Malden, MA, USA, January 2002. ISBN 978-0-470-99665-2 978-0-631-21840-11 doi: 10. 00.102999/8447096622.c.11
Arnoud De Kroon, Joris Mooij, and Danielle Belgrave. Causal Bandits without prior knowledge using separating sets. In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 407-427. PMIRR, June 2022.
38

---

Published in Transactions on Machine Learning Research (11/2023)
Kun Kuang, Lian Li, Zhi Geng, Lei Xu, Kun Zhang, Beishui Liao, Huaxin Huang, Peng Ding, Wang Miao, and Zhichao Jiang. Causal Inference. Engineering, 6(3):253-2)) 223,, March 2020. ISSN 2095-8099. doi: 00.1016/j.eng:2011....0000
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179 1191, 2020.
Daniel Kumor, Junzhe Zhang, and Elias Barcinboim. Sequential Causal Imitation Learning with Unobserved Confounders. In Advances in Neural Information Processing Systems, volume 34, pp. 14669 14680. Curran Associates, Inc., 2021.
Matt J. Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Marc Lanclot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A Unified Game-Thhoree Approach to Multiagent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Finnian Lattimore, Tor Lattimore, and Mark D. Reid. Causal bandits: Learning good interventions via causal inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 1189 1197, Red Hook, NY, USA, December 2016. Curran Associates Inc. ISBN 978-1-5108-3881-9
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network Randomization: Simple Technique for Generalization in Deep Reinforcement Learning. In International Conference on Learning Representations, March 2020a.
Sanghack Lee and Elias Bareinboim. Structural Causal Bandits: Where to Intervene? In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Sanghack Lee, Juan D. Correa, and Elias Bareinboim. General Identifiability with Arbitrary Surrogate Experiments. In Proceedings of The 35th Uncerlainly in Artificial Intelligncce Conference, pp. 389-398. PMLR, August 2020b.
Tabitha E. Lee, Jialiang Alan Zhao, Amrita S. Sawhney, Siddharth Girdhar, and Oliver Kroemer. Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policies. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4776-4782, May 2021. doi: N..1099/CRA485000 202...6612.1111333
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. Xi preprint arXiv:2005.01644 2020.
Luofeng Liao, Zuyue Fu, Zhuoran Yang, Yixin Wang, Mladen Kolar, and Zhaoran Wang. Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning, July 2021.
Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pp. 1150-3158.. PMLR, 2018.
David Lopez-Pazz Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Leon Bottou. Discovering Causal Signals in Images. In 2017 7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 58 66, Honolulu, HI, July 2017. IEEE. ISBN 978-1-5386-0457-1. doi:
Chaochao Lu and José Miguel Hernández Lobato. Deconfounding Reinforcement Learning in Observational Settings. arXio preprind ar 181: 10576, December 2018.
Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-oobto, Kun Zhang, and Bernhard Schölkopf. Sample-Efficiet Reinforcement Learning via Counterfactual--Beed Data Augmentation, December 2020.
Yangyi Lu, Amirhossein Meisami, and Ambuj Tewari. Causal Bandits with Unknown Graph Structure. Advances in Neural Information Processing Systems, October 2021.
39

---

Published in Transactions on Machine Learning Research (11/2023)
Yangyi Lu, Amirhossein Meisami, and Ambuj Tewari. Efficient Reinforcement Learning with Prior Causal Knowledge. In Pirst Conference on Causal Learning and Reasoning, February 2022.
Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A Survey on Model-basee Reinforcement Learning, June 2022.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable Reinforcement Learning through a Causal Lens. Proceedings of the AAAT Conference on Artificial Intelligence, 34((3))24433-255) 2500, doi: 10. 16009//aai.v3ii000
April 2020. ISSN 2374-3468.
Edwin P. Martens, Wiebe R. Pestman, Authonius de Boer, Svetlana V. Belitser, and Olaf H. Klungel. Instrumental Variables: Application and Limitations. Epidemiology, 17(3):260-267, 2006. ISSN 1044- 3983. doi: 10.1097
Christopher Mock. Causal inference and causal explanation with background knowlodge. In Proceedings of the Eleventh Conference on Uncerlainly in Arlificial Intelligence, UAL95, pp. 403-410 4314, San Francisco, CA, USA, August 1995. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-385-1.
Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Thomas S. Stepleton, Nicolas Heess, Arthur Guez, Eric Moulines, Marcus Hutter, Lars Buesing, and Remi Munos. Counterfactual Credit Assignment in Model-Free Reinforcement Learning. In Proceedings of the 38th Intervational Conference on Machine Learning, pp.
PMLR, July 2021.
Wang Miao, Zhi Geng, and Eric Tchetgen Tchetgen. Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder. Biometrika, 105(4):987-995) -933, December 2018. ISSN 0006-3444. doi: 1..1093/biome//asy//n
Eduardo Mosqueira-Rey, Flena Hernández-Pereira, David Alonso-Ríos José Robes-Bascaree and Ángel Fernández-Less Leal. Human-in-the-loww machine learning: A state of the art. Artificial Intelligence Review, August 2022. ISSN 1573-7462. doi: /tt///102222-2221222222
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32, 2019.
Susan A. Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Sociely: Series B (Statistical Methodology), 65(2):331 -5551,3555551 2003. doi: 11111111467-958800003
Razich Nabi, Daniel Malinsky, and Ilya Shpitser. Learning Optimal Fair Policies. In Proceedings of the 36th International Conference on Machine Learning, pp. 4674-4682. PMTR, May 2019.
Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal Induction from Visual Observations for Goal Directed Tasks, October 2019.
Vineet Nair, Vishakha Patil, and Gaurav Sinha. Budgeted and Non- Son-Budgeted Causal Bandits. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 2017-2025. PMLR, March 2021.
Hongscok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding. Advances in Newral Information Processing Systems, volume 33, pp.
Curran Associates, Inc., 2020.
Michael Oberst and David Sontag. Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models. In Proceedings of the 36th International Conference on Machine Learning, pp. 4881-4890. PMLR, May 2019.
OpenAl, llge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving Rubik's Cube with Robot Hand, October 2019.
40

---

Published in Transactions on Machine Learning Research (11/2023)
Ian Osband, Daniel Russo, and Benjamin Van Roy. (More) Efficient Reinforcement Learning via. Posterior Sampling, In Advances in Newral Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, October 2022.
Alizée Pace, Hugo Yeche, Bernhard Schölkopf, Gunnar Rätsch, and Guy Tennenholtz. Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding, June 2023.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning. arXiv preprint arXie:1511 (tfe:::5111003333 06342, November 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. crriosity-driver Exploration by Self- supervised Prediction. In Proceedings of the 34th International Conference on Machine Learning, pp. 2778-2787. PMLR, July 2017.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669 688, December 1995. ISSN 0006-3444. doi: 10.1093/biomett (80mee/82.4.669.
Judea Pearl. Causal inference in statistios: An overview. Stalistics surveys, 3:96 146, 2009a. doi: 10.1214/
09-SS057.
Judea Pearl. Causality. Cambridge university press, 2009b.
Judea Pearl and Elias Bareinboim. External Validity: From Do-Caleulus to Transportability Across Popu- lations. Statistical Science, 29(4), November 2014. ISSN 0883-4237. doi: 10.1214/14-STS486.
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Couse and Effect. Basic books, 2018.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 3803-3810. IEFE, 2018. doi: 10. 10.1109/TCRA.100 0000///CAA.2188.866000 2018.8460588..
Emilija Perkovic, Identifying causal effects in maximally oriented partially directed acyclic graphs. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 530-539. PMLR,
August 2020.
Emilija Perkovic, Markus Kalisch, and Marloes H. Maathuis. Interpreting and using CPDAGs with back- ground knowledge. In Proceedings of the 2017 Conference on Uncertainty in Artificial Intelligence (UA12017), pp. ID: 120. AUAI Press, 2017.
Ronan Perry, Julius Von Kügelgen, and Bernhard Schölkopf. Causal Discovery in Heterogencous Environ- ments Under the Sparse Mechanism Shift Hypothesis. In Advances in Neural Information Processing Systems, October 2022.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 078-0-262-03331-0 978-0-262-34429--
Silviu Pitis, Flliot Creager, and Animesh Garg. Counterfactual Data Augmentation using Locally Factored Dynamics. In Advances in Neural Information Processing Systems, volume 33, pp. 3976-3990. Curran Associates, Inc., 2020.
Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. MoCoDA: Model-based Counterfactual Data Augmentation. In Advances in Neural Information Processing Systems, October 2022.
41

---

Published in Transactions on Machine Learning Research (11/2023)
Drago Plecko and Elias Bareinboim. Causal Fairness Analysis, July 2022.
Drago Plecko and Elias Barcinboim. Causal Fairness for Outcome Control, June 2023.
Erika Puiutta and Eric Veith. Explainable reinforcement learning: A survey, In International Conference for Machine Learning and Knowledge Extraction, pp. 77-95. Springer, 2020. doi: 10.1007/ 5.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep Reinforcement Learning for Sepsis Treatment, November 2017.
Maithra Raghu, Alex Irpan, Jacob Andreas, Bobby Kleinberg, Quoc Le, and Jon Kleinberg, Can deep reinforcement learning solve Brroo--Selridge-SSeenn games? Learning, pp. 4238 4246. PMLR, 2018.
In International Conference on Machine
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization reinforcement learning. In International Conference on Machine Learning, pp. 8787 8798. PMLR, 2021.
Danilo J. Rezende, Ivo Danihelka, George Papamakarios, Nan Rosemary Ke, Ray Jiang, Theophane We- ber, Karol Gregor, Hamza Merzic, Fabio Viola, Jane Wang, Jovana Mitrovic, Frederic Besse, Ioannis Antonoglou, and Lars Buesing. Causally Correct Partial Models for Reinforcement Learning, February 2020.
Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974. doi: 10.1037/h0033350.
Richard M. Ryan and Edward L. Deci. Intrinsic and Extrinsic Motivations: Classic Definitions and New
Directions. Contemporary Educational Psychology, 25(1):54-61 January 2000. ISSN 0361-476X. doi: 10.1006/ceps..199..00.
Sorawit Saengkyongam, Nikolaj Thams, Jonas Peters, and Niklas Pfister. Invariant Policy Learning: A Causal Perspective, September 2022.
Kristin L. Sainani. Instrumental Variables: Uses and Limitations. PMOR, 10(3):303 308, March 2018. ISSN 1934-1482. doi: p0.1016/1.pmmr.2208822222
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):
612-6344 2021. doi: 10.1109/PRRO..2021..5555
Carolyn A Schult and Henry M Wellman. Explaining human movements and actions: Children's understand- ing of the limits of psychological explanation. Cognition, 62(3):291 324, March 1997. ISSN 0010-0277. doi: 10. 10016//00000007770)00788833333
Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal Influence Detection for Improving Efficiency in Reinforcement Learning. In Advances in Neural Information Processing Systems, October 2021.
Rajat Sen, Karthikeyan Shanmugam, Alexandros G. Dimakis, and Sanjay Shakkottai. Identifying Best Interventions through Online Importance Sampling. In Proceedings of the 34th International Conference on Machine Learning, pp. 3057 3066. PMLR, July 2017.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Weakly Supervised Disentangled Generative Causal Representation Learning. Journal of Machine Learning Research, 23 (241):1-55, 2022. ISSN 1533-7928.
Zheyan Shen, Peng Cui, Kun Kuang, Bo Li, and Peixuan Chen. Causally Regularized Learning with Agnostic Data Selection Bias. In Proceedings of the 26th ACM International Conference on Multimedia, pp. 411-419, October 2018. doi: 10.1145/3240508.33 3240077.3
42

---

Published in Transactions on Machine Learning Research (11/2023)
Ily Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semi-Markovian causal models. In Proceedings of the National Conference on Artificial Intelligence, volume 21, pp. 1219. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006.
Πya Shpitser and Judea Pearl. What counterfactuals can be tested. In Proceedings of the Twenty- Twenty-Third Third
Conference Uncertainty in Artificial Intelligence, UAI'07, pp. 352 359. AUAI Press, July 2007.
Thomas R. Shultz. Rules of Causal Attribution. Monographs of the Society for Research in Child Develop- ment, 47(1):1 51, 1982. ISSN 0037-976X. doi: 10.2307/1165880
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-3355 -599, October 2017. ISSN 1476-4687. doi: 10.1038/ 10.1038/naturee n288//a2aur22200.
Steven Sloman. Causal Models: How People Think about the World and Its Alternatives. Oxford University Press, 2005.
Steven A. Sloman and David Lagnado. Causality in Thought, Annual Review of Psychology, 66(1):223 247, 2015. doi: 10.1146/ammnt http//aannsssss----//o (110 010814-015135.
David Sobel and Jessica Sommerville. The Importance of Discovery in Children's Causal Learning from Interventions. Frontiers in Psychology, 1, 2010. ISSN 1664-1078. doi: 00.3389/fpsyg.2011 2010.0017.
Sumedh A. Sontakke, Arash Mehrjou, Laurent Itti, and Bernhard Schölkopf. Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning. In Proceedings of the 38th International Conference on Machine Learning, pp.
PMTR, July 2021.
Peter Spirtes, Clark N. Glymour, Richard Scheines, and David Heckerman. Causation, Prediction, and Search. MIT Press, 2000. ISBN 978-0-262-19440-2.
James H. Stock and Francesco Trebbi. Retrospectives: Who invented instrumental variable regression?
Journal of Economic Perspectives, 1733):177-194, 2003. doi: 10. 10.1257 101137//8663330577604116
Jianyu Su, Stephen Adams, and Peter A. Beling. Counterfactual Multi-Agent Reinforcement Learning with
Graph Convolution Communication, December 2020.
Xiaohai Sun, Dominik Janzing, Bernhard Schölkopf, and Kenji Fukumizu. A kernel-based causal learning algorithm. In Proceedings of the 24th International Conference on Machine Learning, ICML '07, pp. 855- 862, New York, NY, USA, June 2007. Association for Computing Machinery. ISBN 978-1-595993-993-3 doi: 10.1145/12788 (11127799.1776601
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Gokul Swamy, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Causal Imitation Learning under Tem- porally Correlated Noise. In Proceedings of the 39th International Conference Machine Learning, pp- 20877 20890. PMLR, June 2022.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiasod Scene Graph Gen- eration From Biased Training. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 3713-300 17222 Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817-168-5. doi: http:///2R2200.2220000000
Zeyu Tang, Yatong Chen, Yang Liu, and Kun Zhang. Tier Balancing: Towards Dynamic Fairness over Un- derlying Causal Factors. In The Eleventh International Conference on Learning Representations, February 2023.
Yee Teh, Victor Bapst, Wojeiech M. Czarnocki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforeement learning. In Advonces in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
43

---

Published in Transactions on Machine Learning Research (11/2023)
Nikolaj Thams. Causality and Distribution Shift. PhD thesis, University of Copenhagen, 2022.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 TEFE/RS /RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23 30. IEEE, 2017. doi: 10.1109/ 100S.2017.820222
Thien Q. Tran, Kazuto Fukuchi, Youhei Akimoto, and Jun Sakuma. Unsupervised Causal Binary Concepts Discovery with VAE for Black-Box Model Explanation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(9) 36(9):9614-9)) 996449922, June 2022. ISSN 2374-3468. doi: 10.1609 10099/aaa.vv380 v36i9. 21199.
Stelios Triantafyllou, Adish Singla, and Goran Radanovic. Actual Causality and Responsibility Attribution in Decentralized Partially Observable Markov Decision Processes. In Proceedings of the 2022 AAAT/ACM Conference AI, Ethics, and Society, AIES '22 pp. 739-752, New York, NY, USA, July 2022. Association for Computing Machinery. ISBN 978-1-4503-9247-11 doi: 10.
Stratis Tsirtsis, Abir De, and Manuel Rodriguez. Counterfactual Explanations in Sequential Decision Making Under Uncertainty. In Advances in Neural Information Processing Systems, volume 34, pp. 30127-30133 Curran Associates, Inc., 2021.
Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan E. Hines, John P. Dickerson, and Chirag Shah. Counterfactual Explanations and Algorithmic Recourses for Machine Learning: Review, November 2022.
Nelson Vithayathil Varghese and Qusay H. Mahmoud. survey of multi-task doep reinforcement learning. Electromics, 999):1363, 2020. doi: 10.3390 10.3390/eeeccronicce eelectronics90913366
Matthew J. Vowels, Necati Cihan Camgoz, and Richard Bowden. D'ya Like DAGs? Survey on Structure Learning and Causal Discovery. ACM Computing Surveys, 55(4):82:1-82233 November 2022. ISSN 0360- 0300. doi:
Jane X. Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, Francis Song, Gavin Buttimore, David P. Reichert, Neil Rabinowitz, Loic Matthey, Demis Hassabis, Alexander Lerchner, and Matthew Botvinick. Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents, October 2021a.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. Advances Neural Information Processing Systems, 53:7968-7978 2020a.
Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably Efficient Causal Reinforcement Learning with Confounded Observational Data. In Advonces Neural Information Processing Systems, volume 34, pp. 21164-21 21155. Curran Associates, Inc., 2021b.
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual Commonsense R-CNN. In 2020 IEEE/CVP Conference on Computer Vision and Pattera Recogrition (CVPR), pp. 10757 10767, Seattle, WA, USA, June 2020b. IEEE. ISBN 978-1-72817-168-5, doi: 10.
Tan Wang, Chang Zhou, Qianru Sun, and Hanwang Zhang. Causal Attention for Unbiased Visual Recog- nition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3091 3100, 2021c
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Picter Abbeel, and Jimmy Ba. Benchmarking Model-Basod Reinforcement Learning, July 2019.
Yixin Wang and Michael I. Jordan. Desiderata for Representation Learning: A Causal Perspective, February 2022.
44

---

Published in Transactions on Machine Learning Research (11/2023)
Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal Dynamics Learning for Task- Independent State Abstraction. In Proceedings of the 39th International Conference on Machine Learning, pp. 23151 23180. PMLR, June 2022.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. Finetuned Language Models Are Zero-Shot Learners. arXiv preprint arXiv:2109 09.01652, September 2021.
Henry M. Wellman. The Child's Theory of Mind. The Child's Theory of Mind. The MIT Press, Cambridge, MA, US, 1992. ISBN 118-0-262-231522 978-0-262-73099--0.
Min Wen, Osbert Bastani, and Ufuk Topcu. Algorithms for fairness in sequential decision making. In International Conference on Artificial Intelligence and Statistics, pp. 1144-1152. PMLR, 2021.
Xiaobao Wu, Chunping Li, and Yishu Miao. Discovering Topics in Long-tailed Corpora with Causal Inter- vention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 175-185, Ouline, August 2021. Association for Computational Linguistics. doi: 10.18653/ / w1t2/22..iiimiiiaee windiiinnndddd acl.ss.acl.1... 15.
Annic Xie and Chelsea Finn. Lifelong Robotic Reinforcement Learning by Retaining Expericnces. In Pro- ceedingg of The 1sl Conference on Lifelong Learning Agents, pp. 838 855. PMLR, November 2022.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-Grradent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Akihiro Yabe, Daisuke Hatano, Hanna Sumita, Shinji Ito, Naonori Kakimura, Takuro Fukunaga, and Ken- ichi Kawarabayashi. Causal Bandits with Propagating Inference. In Proceedings of the 35th International Conference on Machine Learning, pp. 5512 5520. PMLR, July 2018
Chao-Han Huck Yang, I.-Te Danny Hung, Yi Ouyang, and Pin- Yu Chen. Training a Resilient Q-network against Observational Interference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8814-8822, June 2022a. doi: 10000///aai.v668.....
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. CausalVAE: Dis- entangled Reprosentation Learning via Neural Structural Causal Models. In 2021 IEEE/CVE Con- ference on Computer Vision and Pallern Recognition (CVPR), pp. 9588 9597, June 2021a. doi: D0.1109/CVPRA6437.222 202.444477.222.009477.
Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in Deep Reinforcement Learning: A Comprehensive Survey, July 2022b.
Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto Policy Pool for Model-based Offline Reinforcement Learning. In International Conference on Learning Representations, 2021b.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image Augmentation Is All You Need: Regularizing Decp Reinforcement Learning from Pixels. In International Conference on Learning Representations, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A Benchmark and Evaluation for Multi- Task and Meta Reinforcement Learning, June 2021.
Yang Yu. Towards sample efficient reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI118 pp. 5739-55 9739-5743, Stockholm, Sweden, July 2018. AAAT Press. ISBN 975-0-9992411-2277
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fairness Constraints: Mechanisms for Fair Classification, July 2015.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascamu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019.
45

---

Published in Transactions on Machine Learning Research (11/2023)
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. Xrxiv preprinl or Xin: 118800067993 2018a.
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant Causal Prediction for Block MDPs. In Proceedings of the 37th International Conference on Machine Learning, pp. 11214 11224. PMLR, November 2020a.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning Invariant Representations for Reinforcement Learning without Reconstruction. In International Conference Learning Representations, February 2022a.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.0660 2018b.
Junzhe Zhang and Elias Bareinboim. Markov Decision Processes with Unobserved Confounders: A Causal Approach. Technical report, Technical report, Technical Report R-23, Purdue Al Lab, 2016.
Junzhe Zhang and Elias Bareinboim. Transfer Learning in Multi-Armer Bandits: A Causal Approach. In Proceedings of the Twenty-Sirth International Joint Conference on Artificial Intelligence, pp. 1340-1346, Melbourne, Australia, August 2017. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-9992411-0-41 doi: 10.24963/ijcai http://cca..011///// 2017/18..
Junzhe Zhang and Elias Bareinboim. Fairness in Decision-Making The Causal Explanation Formula. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2374-3468. doi: 10.1609 10009//aai.v00 wat.aa........111..1.1
Junzhe Zhang and Elias Bareinboim. Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Junzhe Zhang and Elias Bareinboim. Designing Optimal Dynamic Treatment Regimes: A Causal Reinforce- ment Learning Approach. In Proceedings of the 37th International Conference on Machine Learning, pp. 11012-11022. PMLR, November 2020.
Junzhe Zhang and Elias Bareinboim. Can Humans Be out of the Loop? In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 1010-1025. PMLR, June 2022a.
Junzhe Zhang and Elias Bareinboim. Online Reinforcement Learning for Mixed Policy Scopes. In Advances in Neural Information Processing Systems, October 2022b.
Junzhe Zhang, Daniel Kumor, and Elias Barcinboim. Causal imitation learning with unobserved confounders. In Proceedings of the 34th International Conference on Newral Information Processing Systems, N0PS'20, pp. 12263- 12274, Red Hook, NY, USA, December 2020b. Curran Associates Inc. ISBN 978-1-71382-9544
Junzhe Zhang, Jin Tian, and Elias Barcinboim. Partial Counterfactual Identification from Observational and Experimental Data. In Proceedings of the 39th International Conference on Machine Learning, pp. 26548 26558. PMLR, June 2022b.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Steeet-Seventh Conference on Uncertainty in Artificiol Intelligence, UAI'11, pp. 804-813, Arlington, Virginia, USA, July 2011. AUAI Press. ISBN 978-0-9749039---22
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Niith AAAT Conference on Artificial Intelligence, 2015. doi: 10.1609/aaan waw..................
Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue Hc, and Zheyan Shen. Deep Stable Learning for Duttoof-Dsttrbuttie Generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp.
2021a.
46

---

Published in Transactions on Machine Learning Research (11/2023)
Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? In Advances in Neural Information Processing Systems. volume 33, pp. 18457 18469. Curran Associates, Inc., 2020c.
Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. Causal Intervention for Leveraging Popularity Bias in Recommendation. In Proceedings the 44th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 11-20, New York, NY, USA, July 2021b. Association for Computing Machinery. ISBN 978-1-4503--033 doi:
Yu Zheng, Chen Gao, Xiang Li, Xiangnan He, Yong Li, and Depeng Jin. Disentangling User Interest and Conformity for Recommendation with Causal Embedding. In Proceedings of the Web Conference 2021, WWW '21, pp. 2980-2991, New York, NY, USA, June 2021. Association for Computing Machinery. ISBN 978-1-4503-8312-7. doi: 10.1145/3412381.4442 133497888..
Hanhan Zhou, Tian Lan, and Vaneet Aggarwal. PAC: Assisted Value Factorization with Counterfactual Pre- dictions in Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, October 2022.
Deyao Zhu, Li Erran Li, and Mohamed Elhoseiny. CausalDyna: Improving Generalization of Dyna-style Reinforcement Learning via Counterfactual-Base Data Augmentation. November 2021.
Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal Discovery with Reinforcement Learning. In Interna- lional Conference on Learning Representations, February 2022a.
Zheng-Mao Zhu, Xiong-Huu Hui Chen, Hong-Long Tian, Kun Zhang, and Yang Yu. Offline Reinforcement Learning with Causal Structured World Models, June 2022b.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint 0tt::2009007888 2020.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine- Tinn-Tunigg Language Models from Human Preferences, January 2020.
47

---

Published in Transactions on Machine Learning Research (11/2023)
A Supplementary Introduction to Causality
A.1 Mediation analysis
Mediation is causal concept. that closely relates to counterfactuals. The goal of mediation analysis is to examine the direct indirect effects of treatment variable x on outcome variable Y, mediated by a third variable M (referred to as the mediator). To illustrate, let's consider the direct effect of different food consumption on preventing scurvy (See Fjgurc3). In a counterfactual world, if we prevent changes in the mediator variable M (e.g., vitamin C intake), then whatever changes the outcome variable Y (e.g., whether a sailor gets seurvy) can only be attributed to changes in the treatment variable X (e.g., the food consumed), allowing us to establish the observed effect as a direct effect of X on Y. It worth noting that, in cases like this, the statistical language can only provide the conditioning operator, which merely shifts our attention to individuals with equal values of M. On the other hand, the do-operator preciscly captures the concept of keeping the medialor M unchanged. These two operations lead to fundamentally different results, and conflating them can yield opposite conclusions (Pearl & Mackenzie 2018 Chapter 9). In summary, such analyses are crucial for understanding the potential causal mechanisms and paths in complex systems, with applications spanning various fields including psychology, sociology, and epidemiology (Carey & Wu 2022)
A.2 Causal discovery and causal reasoning
In the field of causal inference, there are two primary areas of focus: causal discovery and causal reasoning. Causal discovery involves inferring the causal relationships between variables of interest (in other words, identifying the causal graph of the data generation process). Traditional approaches use conditional inde- pendence lests to infer causal relationships, and recently some studies have been conducted based on large datasets using deep learning techniques. Glymour et al. (2019) and Vowels et al. (2022) comprehensively
survey the field of causal discovery.
As opposed to causal discovery, causal reasoning investigates how to estimate causal effects, such as in- tervention probability, given the causal model. Interventions involve actively manipulating the system environment, which can be costly and potentially dangerous (c.g., testing a new drug in medical experiments). Therefore, a core challenge of causal reasoning is how to translate causal effects into estimands that can be estimated from observational data using statistical methods. Given the causal graph, the identifiability of causal effects can be determined systematically through the use of do-calculus (Pearl 1995).
A.3 Causal representation learning
A fundamental limitation of traditional causal inference is that most research starts with the assumption that causal variables are given, which does not align with the realily of dealing with high-dinennsooall and low-level data, such as images, in our daily lives, Causal representation learning (Schölkopf et al. 2021) is an emerging field dedicated to addressing this challenge. Specifically, causal representation learning locuses on identifying high-level variables from low-level observations. These high-level variables are not only descriptive of the observed data but also explanatory of the data generation process, as they capture the underlying causal mechanisms. By effectively discovering meaningful and interpretable high-level variables, causal representation learning facilitates causal inference in complex, high-dimensional domains.
B A Brief Introduction to Environments and Tasks
In this section, we briefly introduce the environments and tasks mentioned spanning sections to
9 A treatment variable, also known an intervention variable, refers to variable that is purposefully manipulated to assess impact on one or more outcome variables of interest.
48

---

Published in Transactions on Machine Learning Research (11/2023)
B.1
Autonomous Driving
BARK-ML: https://github. www.bark http://arr-smmlaattrrrrrralall BARK is an open-source behavior bench- marking environment. It covers a full varicty of real- real-world, interactive human behaviors for traffic partici- pants, including Highway, Merging, and Intersection (Unprotected Left Turn).
The highway-env project gathers collection of environments for autonomous driving and tactical decision-making tasks, including Highway, Merge, Roundabout, Parking, Intersection, and Racetrack. Crash is a modified version by (Ding et al. 2022) that is not publicly available.
Crash (highway-env): https://github.
B.2 Classical Control
Cart-polc [dm_control)) attps://githii www.///wkraaaaa/ddnnnnnmno dontrrolloolll Cart-pole is an environment belonging to the DeepMind Control Suite. It involves swinging up and balancing an unactuated pole by applying forces to a cart at its base.
Acrobot (OpenAI Gym):
https///www.glllbarrrr
The Acrobot environment consists of two links connected linearly to form chain, with one end of the chain fixed. The goal is to apply torques the actuated joint to swing the free end of the linear chain above given height while starting from the initial state of hanging downwards.
Mountain Car (OpenAI Gym): https:///ww http:///wwwwgmllbbrr...oooo hww.a//awy....rrrrrrroo/////aaas
mountain_cart car/ The Mountain Car MDP is a deterministic MDP that consists of a car placed stochas- tically at the bottom of sinusoidal valley, with the only possible actions being the accelerations that can be applied to the car in either direction. The goal of the MDP is strategically accelerate the car to reach the goal state on top of the right hill.
Cart Pole (OpenAI Gym): cttp:://ww..gmllibrrayyy wtt.://.iibbaa..d.d///rrrrrrroo hter://wwi.romeettscaassiiii.o., control/catttttt...ooooo pole/ A pole is attached by an un-actuated joint to cart, which moves along a frictionless track. The pendulum is placed upright on the cart and the goal is to balance the pole by applying forces in the left and right direction on the cart.
Pendulum (OpenAI Gym):
control/
pendulum/] The system consists of pendulum attached at one end to fixed point, and the other end being free. The pendulum starts in random position and the goal is to apply torque the free end to swing it into an upright position, with its center of gravity right above the fixed point.
Inverted Pendulum (OpenAI Gym): https://www
www.wwggllbbrrr..mm
This environment involves a carl that can moved linearly, with a pole fixed on it at one end and having another end free. The cart can be pushed left or right, and the goal is to balance the pole on the top of the cart by applying forces on the cart.
pnperted_pendulurg
B.3 Game
MiniPacman:
http:://www...oooggggggg//////aa Agents
MiniPacman is
played in 15 x 19 grrd-world. Characters, the ghosts and Pacman, move through a maze.
Lunar Lander (OpenAI Gym): hnttp:://www..tt. www.ww/agwlbbrrr...mmmmmm....... hwtp.//////iiioooeet////////////
_ander//
This environment is classic rocket trajectory optimization problem. The landing pad is always at coordi-
nates 0). The coordinates are the first two numbers in the state vector. Landing outside of the landing pad is possible. Fuel is infinite, so an agent can learn to fly and then land on its first attempt.
Bipedal Walker (OpenAI Gym): httpss//wwr...oooooooeeee////rar
walker/ This a simple 4-joint walker robot environment.
Car Racing (OpenAl Gym): https://www wwwp:wwwwwwbrr....mm.mow..//...w dev aasr//eniirommeet/boodddd http:///oooreeaaarraaangg/////// Car Racing top-down racing environment. The generated track is random in every episode. Some indicators
49

---

Published in Transactions on Machine Learning Research (11/2023)
are shown at the bottom of the window along with the state RGB buffer. From left to right: true speed, four ABS sensors, steering wheel position, and gyroscope.
Beam Rider (OpenAl Gym); https://www.gy/: gyml www.gygaglbbbr.r.oo. httaaa//w/eee....mettttttttt//// /ttph///bammrrreer//e/ rider/ The agent controls a space-ship that travels forward at a constant speed. The agent can only steer sideways between discrete positions. The goal to destroy enemy ships, avoid their attacks and dodge space debris.
Pong (OpenAI Gym): https://www httpas///wwwgglbbbarrrr http://///eeeeeeeomme///t/////// the right paddle, competing against the left paddle controlled by the computer.
The agent controls
Pong (Roboschool):
http:///ttttb..oooo./aaa////osol Roboschool is an open-source software for robot simulation, which is now deprecated. Pong allows for multiplayer training.
This game is a transportation puzzle, where the player has to push all boxes in the room on the storage locations/ targets. The possibility of making irreversible mistakes makes these puzzles so challenging especially for RL algorithms.
Sokoban: https://github
SC2LE is RL environment based on the StarCraft II game. It is multi-agent problem with multiple players interacting. This domain poses a grand challenge raising from the imperfect information, large action and state space, and delayed credit assignment.
SC2LE: https://github. con/deeppindd/yss2222 https//trrrrrgggtto.o..o///ppprp
VizDoom: https://github. hom//Parama-Fundatioou Viiioooo ViZDoom is based on Doom, a 1993 game. It allows developing AI bots that play Doom using only visual information.
B.4 Healthcare
MIMIC-III: hctps://ppyysionet. org//onnten///inniiii//1. MIMIC-III (Johnson et al. 2016 is a large, freely-available database comprising deidentified health-related data associated with over forty thou- sand patients who stayed critical care units of the Beth Israel Deaconess Medical Center between 2001 and 2012. Lu al. (2020) use the code (https://githul:
pro- vided by RRahu al. (2017) for preprocessing the data and Bica al (2021b) open-source their code http:p/:/hhccccoe.eeaaaaaaaaaaaa
Therapy: /tee
data Bata//therapy/READee This dataset contains real data from cogmitive behavioral therapy. The data were collected during a clinical trial with the patients' written consent A post-processed version of the data is available upon request from Krristina.Furr@nn
moni-tuebinggn.co
This environment, originally developed by Oberst & Sontag] (2019) to simulate intensive care trajectories, is adopted by Pace al.) (2023 to investi- gate the impact of hidden confounders in offline RL. The state space comprises 4-dimensional normalized obsorvation vectors (heart rate, systolic blood pressure, oxygenation, and blood glucose levels), and the discrete action space encompasses three binary treatments (antibiotic, vasopressor, and ventilation). In addition, the patient's diabetic status serves as an unobserved binary variable in this environment.
Sepsis: hhttps://githbb com/ hwtpc///eLn.iaaaallguuuelllllaaa
HiRID: https://physionee http:///pppapaoooo..ooooo//tttta
The HiRID dataset (Hyland et al. 2020 Fal-
tys al., 2021). comprising data from over 34,000 patient admissions at the Bern University Hospital's Department of Intensive Care Medicine in Switzerland, offers high-resolution collection of demographic, physiological, diagnostic, and treatment information. Pace et al. (2023) focused on optimizing fluid and vasopressor administration to prevent circulatory failure in their research targeted at non-identiftable hidden confounding in offline RL.
B.5 Robotics
B.5.1
Locomotion
Cheetah
httpps///githu. com/svikramank/dn conordrcoooooollooooooood Cheetah is an environment
belonging to the DeepMind Control Suite. It. is running planar bipedal robot.
50

---

Published in Transactions on Machine Learning Research (11/2023)
OpenAI Gym: htpp:://www.gymlbbrae. htt.///rww....///oeooooooo/////p
These environments are built upon the MuJoCo (Multi-Joint dynamics with Contact) engine. The goal is to make the 3D robots move in the forward direction by applying torques on the hinges connecting the links of each leg and the torso.
PyBullet Gym: httpp:///gtthb:
comp//obeeeellttppppllleet-t-- http://////gttoooooo//ooooooo/op gym
This is an open-source implementation
of OpenAT Gym MuJoCo environments using the Bullet Physics bullet3).
D4RL: https://github.. http://ow..ooo//raaaaooooooaooaa D4RL is open-source benchmark for offline RL. It includes several OpenAl Gym benchmark tasks, such as the Hopper, HalfCheetah, and Walker environments.
B.5.2 Manipulation
CausalWorld an open-source simula- tion framework and benchmark for causal structure and transfer learning in robotic manipulation environ- ment where tasks range from rather simple to extremely hard. Tasks consist of constructing 3D shapes from given set of blocks inspired by how children learn to build complex structures.
CausalWorld: https://giitubb.com httpe//aw.ccorrr-eeeeeeeanaaaaes
Isaac Gym: https://github..
Isaac Gym offers a high perfor- mance learning platform to train policies for wide variety of robotics tasks directly on GPU.
wtwp:ttww.....omotoott com IWVIIIIIIIIIAAAA wwwpa//iiiieeeeeaaaaaaaaaaa.eae
OpenAI: The origin version is developed by OpenAT, known as "Ingredients for robotics research" (https: and now is maintained by the Farama It contains eight simulated
ftnnniaiiiiitioee//..../ooo///en robotics environments.
robosuite is simulation framework powered by the MuJoCo physics engine for robot learning. It contains seven robot models, eight grip- per models, six controller modes, and nine standardized tasks. also offers a modular design for building new environments with procodural generation.
robosuite: http:://gggtttoo//////ttttatiiit ccam:AAlll-IImtiiatttverrrsbssss
cww/ARISE
B.6 Navigation
http://www.....eacooooooootatttn The Minigrid library contains a collection of discrete grid-world environments to conduct research on Reinforcement Learning. Unlock is task designed by (Ding et al. 2022). which not publicly available.
Unlock (Minigrid):
Agents are trained on group of training contexts and subsequently tested on two distinct sets of testing contexts within this environment. The objective is to assess the extent to which agents have grasped the causal variables from the training phase and can accurately deduce and extend to new (test) contexts.
Sustextual-Gridwollt
Agent- Imangnggatta--t-innnnntiiiooo Imagination This envi- ronment built on the Unity3d game development engine. It contains an agent that can move around. The environment automatically changes to a new view after every episode.
3D Maze (Unity): https:///gttt Mttp.////aaaaaaeetttutt/e////
Spriteworld: https://github.. http://tthuu.ooom/eeemmmmaae wctmm//eepmii//pppteeeeeeeeeeeee of a 2D arena with simple shapes that can be moved freely. The motivation was to provide much flexibility for procedurally generating multi-object scenes while retaining as simple an interface as possible.
Spriteworld is an environment, that consists
(ettooy_txtttttxill There are four designated locations in the grid world. When the episode starts, the taxi starts off at a random square and the passenger is at a random location. The taxi drives to the passenger's location, picks up the passenger, drives to the passenger's destination (another one of the four specified locations), and then drops off the passenger. Once the passenger is dropped off, the episode ends.
Taxi: https://www http:/wwwwmmmi.mmmmmmmm.........
51

---

Published in Transactions on Machine Learning Research (11/2023)
B.7 Others
By allowing arbilrary causal graphs, this environment facilitates studying complex causal structures of the world. This is illustrated through simple chemical reactions, where changes in onc element's state can cause changes in the state of another variable.
Chemical:
https://gitthbs http:///ww..oooooaaaaaaaaaaaaaar
Light: http:////gtttto..ccmm////stttttt http://////cauaaaannnnnttttiooi iinduciiinnnnnn It consists of the light switch environment for studying visual causal induction, where N switches control N lights, under various causal structures. Includes common cause, common effect, and causal chain relationships.
MNIST:
http://yann MNIST:httt:///aan ten digits.
lecun. wwtpeedbbe/ssst//////oco
The MNIST datasct contains 70,000 images of handwrit-
52


Reviews:
[{"review_id": "u6YH61BC3Z", "reviewer": "Reviewer_H8ZM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n - the survey compiles a wide range of existing works in causal RL, which is a significant contribution in itself\n - the problem-oriented categorization of these works also makes sense and I believe can be useful to newcomers and practitioners\n\n### Weaknesses\n\n - the introduction (Section 1) is problematic. I find it is very poor at motivating what causality can bring to the field of RL, and is in fact quite misleading. It does not even mention the requirements and restrictions of most causal RL approaches (prior causal knowledge), but rather presents all of the beliefs of the field (\"causality can significantly improve the robustness and interpretability of machine learning models\") as indisputable facts. Such a one-sided view is ok for an opinion paper, but it is particularly problematic for a survey paper aimed at newcomers. Instead of providing a fair overview of the promise, challenges and open questions, it perpetuates the delusion that causality is a magic tool that will solve all of RL's problems, which is in the end detrimental to the field. See my detailed comments.\n - the background (Section 2) contains technical ambiguities and inconsistencies (definition of causal graph, initial state as exogenous variable). It proposes the MDP setting as a unified framework to discuss causal RL approaches, which makes little sense as this setting alleviates most of the difficulties causal RL approaches try to tackle, such as confounding or transfer to new tasks (see my detailed comments). \n - the sections \"Why causality helps [...]\" are too assertive, in the sense that they state \"causality helps\" (indisputable fact) \"here is why\", without any nuance (see my comments on Section 3.2). Again, presenting such strong assertions without nuance about the limitations in a survey paper is I think harmful.\n - the structure of the paper is redundant and confusing at times (e.g., 6.1.1 and 6.4.1, see detailed comments)\n - the content of the section \"Open Problems and Future Directions\" does not fit the description, as it mostly enumerates further existing works that do not fit the proposed taxonomy. I'd suggest to re-work that section and better sort what are actual open problems and future directions. Even better, a section named \"Limitations and future directions\" would make more sense, as in general all of the presented problems (sample efficiency, transfer, spurious correlation, explainability, fairness, safety) are still open problems.\n\nI have also noted some relevant recent works on causal RL that are missing from the survey:\n - Gun et al. ICML 2022 Provably Efficient Offline Reinforcement Learning for Partially Observable Markov Decision Processes\n - Pace et al. ICML 2023 Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding\n - Gasse et al. TMLR 2023 Using Confounded Data in Latent Model-Based Reinforcement Learning\n\n### Detailed comments\n\n - p1 \"emerge\" -> typo\n\n - p2: \"causality can significantly improve the robustness and interpretability of machine learning models\" -> I find this whole paragraph rather ambiguous and not well motivated. The difference between \"pure data-driven\" methods, \"causality\" and \"causal inference\" is not clear. In the proposed example with scurvy, a classic online RL algorithm such as REINFORCE or Q-learning does solve the problem optimally, without the explicit need for causality. The problem here relies on the data at hand, experimental data versus observational data. Causality offers a framework / a set of tools to express this problem, but not necessarily a way to solve it. Based on purely observational data, causal inference is in general impossible. I find that this paragraph does a very poor job at explaining the essence of the problem, and rather lures readers into the delusion that causality is a generic tool that can just be applied on top of any ML model, and will improve their robustness as by magic. This is especially problematic for a survey paper targeted at newcomers in the field.\n\n - p2 \"in most RL studies, agents are only allowed to intervene in action variables\" -> why is this a problem? For which purpose is this a problem? The general goal in RL is to find an optimal policy for an agent that can act only using the action variables. What would be the benefit of knowing what would happen if one could act on other variables, if in the end these variables are not actionable? It seems like the narrative here is \"RL needs causality because RL can not learn causality\". What is missing here is, why would a full causal model benefit RL in the first place? Again, I find this paragraph unconvincing and very poor at motivating what causality can bring to existing RL algorithms.\n\n - p2 \"In contrast, with causal RL, you would begin by analyzing the causal relationships and making informed assumptions\" -> Same comment as before. This example is misleading. Traditional (online) RL requires no assumptions and leads to an optimal policy, given enough data is collected. Causal RL as described here requires either causal assumptions, which can be wrong (superstition, wrong beliefs), or causal knowledge, which in turn requires experimental data to acquire. Causal reasoning on top of wrong assumptions can be harmful. This is a key limitation to all causal RL methods, and deserves to be part of the discussion. Again, I find this narrative quite confusing, very one-sided, and misleading towards the belief that causal inference always brings benefits to RL.\n\n - p3 \"the essence, implications, and opportunities of causal RL\" -> what about the limitations?\n\n - p4 \"definitionn\" -> typo\n\n - p4 \"when the parent set PA(Vi) is too small\" -> I don't see how small parent sets might violate a Markovian property\n\n - p4 \"the omitted variables influence multiple variables in V\" -> what is an omitted variable? Is it the same as an exogenous variable?\n\n - p4 \"Markovian property\" -> has not been defined yet. Which Markovian property?\n\n - p4 \"latent variables\" -> are latent variables equivalent to exogenous variables in your SCM formulation? I find it confusing to refer to exogenous, omitted and latent variables without clarification.\n\n - p5 Figure 3a -> I find this representation of probability distributions hard to read. How do these pie charts represent probability distributions?\n\n - p5 \"sailors would be\" -> would have been\n\n - p8 figure 4 -> In the POMDP setting the policy does not have access to the full state, and can not be represented as $do(a \\sim \\pi(\\dot | s_t))$. POMDPs require a different formalism for the policy. As a suggestion, it would be simpler to present only the POMDP regime, which is more general, and derive the MDP regime as a special case (with constraint $O_t=S_t$). Also, one of the main problems that causality tries to address is confounding, which does not happen in the MDP setting. One more reason to focus on POMDPs.\n\n - p8 \"initial state can be considered an exogenous variable\" -> I do not follow this reasoning. How can the state, reward and action be endogenous variables, and $S_0$ be an exogenous variable at the same time? Is $S_0$ observed or not to decide on the initial action $A_0$? Is this the MDP or the POMDP setting?\n\n - p9 \"On-policy RL directly learns the causal effects of actions on the outcome from interventional data.\" -> Then is on-policy RL already a causal RL method? Is causality only useful in the offline setting?\n\n - p9 \"Fortunately [...] we can identify the desired causal effect through certain causal reasoning techniques\" -> this is not always true (identifiability problem). This sentence is misleading.\n\n - p10 \"Why Causality Helps Improve Sample Efficiency\" -> The paper's title suggests a survey paper, but this section really sounds like an opinion piece. The general tone is 'causality improves sample-efficiency, without question, and here is why'. This is a strong statement, which I find is put forward without due caution. I understand the belief expressed in this section, but I find it problematic that it is presented here as a general truth. The usefulness of causal approaches for sample-efficiency, and for RL in general, yet remains to be demonstrated in the general setting, both in theory and in practice. I strongly suggest changing the tone in this section, and add more nuance, e.g., \"How can causality improve sample efficiency?\"\n\n - p10 \"causality helps reduce the complexity\" -> This is the motivation for a family of approaches, not a general truth. It should be presented as such. \"most / many causal RL approaches are motivated by the idea that exploiting the true causal structure of the problem will reduce the complexity of learning, and thus improve sample efficiency\"\n\n - p10 \"non-causal methods often demonstrate lower efficiency compared to causal RL methods\" -> Again, this assertion is not a universal truth. I'd suggest a less assertive tone: \"in some controlled settings, non-causal methods have been shown to demonstrate lower efficiency compared to causal RL methods\", or \"(Seitzer et al., 2021) show that methods exploiting causal influence for exploration improve sample-efficiency in robotic manipulation tasks\"\n\n - p10 Section 3.3 -> I like this section, which gives a comprehensible summary of existing works. The sub-sections however are highly redundant to the discussion in Section 3.2. I would suggest to merge Sections 3.2 and 3.3 for readability, or simply remove Section 3.2 all together. \n\n - p22 Section 6 \"Considerations Beyond Return\" -> I find the structure of this section confusing. 6.1.1 \"Leveraging Causality for Explainability\" is redundant with 6.4.1 \"Explainable Reinforcement Learning with Causality\", then 6.2.2 \"Leveraging Causality for Fairness\" is redundant with 6.4.2 \"Fair Reinforcement Learning with Causality\", same with 6.3.2 and 6.4.3 about Safety. I suggest to merge these sections.\n\n - p27 Section 7 -> This section is named \"Open Problems and Future Directions\", but in reality it is a continuation of the survey of existing works. How is \"7.2 Causality-aware Multitask and Meta Reinforcement Learning\" an open problem, while \"4.3.1 Generalize to Different Environments\" or \"4.3.2 Generalize to Different Tasks\" are not? This structure does not maker much sense to me.\n\n - p31 \"causal RL is a promising method\" -> Causal RL is not a single method, as stated in definition 2.3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is entitled \"Causal Reinforcement Learning: A Survey\". It introduces some background concepts on causality and RL, presents a broad definition of the umbrella term causal RL, and presents an overview of existing works grouped by the challenges they try to address: sample efficiency, transfer, spurious correlation, and other considerations (explainability, fairness, safety). The paper concludes with open problems and future directions for the field.\n\nThe paper's main contribution is:\n - a broad survey of existing works\n - establishing connections via a problem-oriented taxonomy", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n\n - rework the narrative, in particular the introduction, and present a more nuanced view of causal RL with a less assertive tone\n - fix the technical ambiguities and inconsistencies\n - make it clearer what are the motivations (hopes, beliefs) for causal RL, what has been achieved and demonstrated so far (successes), what are the costs and limitations of causal RL, and what remains to be done\n\nStrengthen the work:\n\n - improve the structure of the paper\n - add missing works to the survey", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YoTneun2vb", "reviewer": "Reviewer_6Lf7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper attempts to provide a catalog of existing algorithms in the field of causal reinforcement learning. This is quite an undertaking since CRL is undergoing rapid development, and a common solution pattern is still emerging. The authors managed to identify some key challenges facing standard RL, which could be addressed using causal inference techniques. A comprehensive list of related work is provided. \n\nStill, there are some questions and confusion regarding the authors' proposed categorization. Some RL researchers and practitioners might find this paper a bit abstract and confusing. The presentation and organization of the manuscript could be improved. More specifically, I summarize my concerns below.\n\n- The need to adopt causal inference techniques in reinforcement learning tasks could be better motivated. It is unclear how existing RL algorithms are unable to address the challenges described in this paper and how causal inference allows one to overcome these challenges. I would recommend the authors starting by describing basic models (e.g., MDPs) and algorithms for RL and providing simple examples, one for each, demonstrating how these RL algorithms do not lead to satisfactory performance when some critical assumptions are violated, e.g.,  the presence of unobserved confounding, selection bias, and generalizability to new environments. \n\n- The categorization within each section could be confusing. For instance, the \"Category\" column in Table 2 seems a bit arbitrary, and some concepts are unclear. What does the \"Mixed\" category mean here, and how does it differ from \"Dynamics\" and \"Tasks\"? Also, there are overlaps between the second row, \"MDP: OPE\" and the third-row, \"POMDP.\" The backdoor/frontdoor adjustment could be seen as a form of off-policy policy evaluation by adjustment. How do they differ, for instance, from the bounding approach in (Namkoong et al., 2020)? Overall, this paper could be most improved by consistently categorizing algorithms across sections. I recommend organizing algorithms according to their problem settings, e.g., off-policy (offline) learning, online learning, offline-to-online (transfer) learning, and imitation learning. The authors could then further categorize algorithms according to their techniques and assumptions.\n\n- Discussions in Section 3 could be improved. It is unclear what type of sample efficiency the authors are referring to. Is it about minimizing the regret in online learning? Or is it about \"zero-shot generalization ability and require only a small number of training samples to converge in new environments?\" As for the latter case, how does it differ from Section 4? Are some of these discussions in Section 4 redundant? There exist methods exploiting the sparsity of the underlying causal relationships to accelerate online learning processes. Overall, I found those algorithms similar to algorithms for factored MDPs in the RL literature. The ability to exploit the underlying independence in the interventional distributions is not unique to causal inference but also exists in other disciplines, such as hierarchical RL and influence diagrams. Due to these reasons, I suggest moving Section 3 to later parts of the manuscript, e.g., Section 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive summary of recent developments in an emerging discipline of causa reinforcement learning (CRL). In other words, CRL studies applying causal inference theory to reinforcement learning algorithms, incorporating prior knowledge about the underlying causal mechanisms. This allows the AI systems/agents to address various challenges and biases arising from the underlying data-generating processes, including the presence of unobserved confounding, selection bias, and generalizability of policies to new environments where external validity does not hold. The authors provide a catalog of CRL algorithms addressing these challenges. More specifically, Section 3 discusses methods to improve the sample efficiency of existing RL algorithms by exploiting underlying sparse state representation; Section 4 describes CRL algorithms to generalize the effects of candidate policies to different environments and tasks; Section 5 studies the challenges of spurious correlations to existing RL algorithms, including unobserved confounding and selection bias. Finally, Section 6 discusses related topics beyond policy optimization, including explainability and algorithmic fairness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- [C1] Add simple examples demonstrating the challenges of unobserved confounding, selection bias, and generalizability to new environments.\n- [C2] Organize algorithms in each section according to their problem settings, such as off-policy, online, offline-to-online (transfer), and imitation learning.\n- [C3] Further clarify the meaning of \"sample efficiency\" in Section 3. Is it referring to online or offline learning? If it is offline, how does it differ from the discussion in Sections 4 and 5?\n- [C4] Generalizability of policies to new environments has been studied in the causal inference literature under the rubrics of transportability. Several conditions and algorithmic procedures have been introduced. The authors should include a few paragraphs discussing related works. See (Bareinboim & Pearl, PNAS-16) for a detailed survey.\n\nMinor comments:\n- [C5] The authors could be more gentle when introducing the causal Markov factorization in Eq. (1). It relies on Markovianity. It does not necessarily hold when unobserved confounders exist in the underlying causal model. I recommend starting with a general factorization, including the exogenous parents $U_i$, and then deriving the Markov factorization when Markovianity holds.", "broader_impact_concerns": "This paper provides a comprehensive survey of existing algorithms and methods exploring the intersection between causal inference and reinforcement learning. No new algorithm is proposed. Its longterm societal impact is unclear to see.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XAwGdqrZhw", "reviewer": "Reviewer_64gv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have a number of suggestions that may be considered. Some of these are mistakes that need to be fixed, and there is a number of them. To contextualize, I am somewhat familiar with the RL literature, and quite familiar with the causality literature:\n\n  * (C1) The Book of Why (TBOW) is cited repeatedly across the paper. This is fine, but I do wish to remark that the Causality book is the canonical reference here. TBOW was intended as a less formal treatment for readers from applied sciences. I think pointers to appropriate Chapters in Causality would be more useful than citations of TBOW.\n  \n  * (C2) The scurvy example in the second paragraph is somewhat difficult to comprehend. From the current writing, my reading is that \"knowing how to treat a disease requires knowing what causes it\", which seems somewhat of an obvious statement. I believe this paragraph could be improved, in particular highlighting how various sentences relate to a formal causal model.\n  \n  * (C3) Relating to (C2), I also think paragraph on the distinction between RL and Causal RL could be improved (page 2, 2nd paragraph). It seems that CRL reduces the size of the action space that is interesting for exploration. However, in my view, the scurvy example currently does not convey this message very clearly.\n  \n  * (C4) Footnote in the SCM paragraph: It is really unusual to state that the Markovian assumption is considered a consensus in causal inference. This is genuinely not the case, witnessed by the huge amount of literature on Semi-Markovian models, and inference in presence of latent confounding. This footnote should be removed altogether.\n  \n  * (C5) Product decomposition paragraph: V_i is not independent of V \\setminus {V_i, pa(V_i), ch(V_i)} given pa(V_i). There could easily be descendants of V_i that are not its children, and conditioning on pa(V_i) does not block any paths to the descendants. This sentence is unsual, and sounds like a description of Markov blanket, which is not relevant here. This part of the manuscript should be revised.\n  \n  * (C6) Distributions P(v_i | pa(v_i)) are usually called Markov kernels. Causal mechanisms are usually reserved for the structural functions f_i. Please double check the Causality book.\n  \n  * (C7) Interventions paragraph: the notion of a submodel appears in counterfactuals. Interventional distributions are defined through submodels. This should be brought forward.\n  \n  * (C8) Interventions paragraph: The common notation is P(Y | do(X = x)), not P(Y | do(X) = x).\n  \n  * (C9) The counterfactual reasoning description is quite cursory (which is perhaps due to considerations of space). However, somewhere, the authors need the emphasize the wildly different nature of quantities that are counterfactuals. The authors give the example of the probability of necessity (PN), while many other more canonical types are usually considered. A good example is the effect of treatment on the treated (ETT). PN is almost never identifiable, while ETT is under lack-of-confounding assumptions.\n  \n  * (C10) Typo: Bellman optimally equations -> Bellman optimality equations.\n  \n  * (C11) Typo: Rl methods -> RL methods.\n  \n  * (C12) Definition of Causal RL -> isn't it also the case that Causal RL sometimes changes the objective of interest? Here I am thinking of counterfactual randomization ideas from [2]. That is, sometimes the ETT-type of quantity may be optimized instead of a total effect.\n  \n  * (C13) page 9, paragraph \"We note ...\": Here again there is a reference to TBOW. A more clear discussion on the difficulties of learning various elements of the causal structure would be desirable. A discussion on various quantities that can be inferred according to Layers 1, 2, and 3 of the causal hierarchy can be found in [1]. Echoing some of these discussion would be rather useful.\n  \n  * (C14) page 11: \"using clustering outcomes as a causal representation\" -> it would be desirable to expand here slightly. From the current writing it is unclear if this is really a causal method or whether it is just a dimensionality-reduction method (there are plenty of non-causal ones). \n  \n  * (C15) Table 1: I wonder how useful the last column of the table is, given it is not discussed in the text as I understand. For example, MIMIC-III appears. This is a super-broad hospital / ICU dataset, and I am not sure how useful a pointer to this is, in the current writing. Perhaps references to the table can be added, or the content adapted.\n  \n  * (C16) page 12: Abduction-Action-Prediction (AAP) -> throughout the text, counterfactual reasoning is used a single block. The number of different quantities that exist within Layer 3 of the causal hierarchy is vast. For some of them, AAP can only be performed if the true underlying SCM is available. Such requirements are incredibly strong, since SCM noise parameterizations are not unique, and cannot be inferred from any data (see [1] again perhaps). These considerations should be made explicit.\n  \n  * (C17) The limitations section does not discuss identifiability issues of counterfactuals. This is one of the main limitations of this approach, and should be stated as such.\n  \n  * (C18) For the part on generalizability: An interesting part of closely related literature is that on transportability [3]. Transportability is a graphical approach that encodes exactly which mechanisms of the SCM remain unperturbed in a new environment, and provides rules for who such invariances could be leveraged. Definitely worth mentioning here, I believe.\n  \n  * (C19) page 19, Addressing Confounding Bias: Valtorta and Shpitser references prove completeness of do-calculus for _inferring Layer 2 quantities from observational data_. A subsequent paper [4] proves completeness for Layer 3 quanties from obs. data. Finally, recent papers [5, 6] show completeness for Layer 2 / 3 from a combination of observational and experimental data. The latter works seem quite related, since RL may often have observational and experimental data simultaneously.\n  \n  * (C20) page 21, last paragraph: this part discussed combining observational and experimental data. I believe this theme could be highlighted as an idea in itself. It features prominently in the causal inference literature, and maybe deserves highlighting in the review as well.\n  \n  * (C21) Addressing Selection Bias (SB): There is a good amount of work on addressing selection bias using graphical models [7]. I am surprised this work is not mentioned, since the tools may well be useful for a Causal RL setting.\n  \n  * (C22) Does the HER setting really constitute SB? The described phenomenon of changing the reward distribution sounds different than what is usually considered SB in the causal literature.\n  \n  * (C23) page 23, last paragraph: TBOW reference appears again for counterfactual reasoning. Furthermore, Kusner and Zhang references are mentioned for fairness, which is fine. However, a number of other results have appeared since, some directly related to fairness in decision-making settings. For example, in the context of causal bandits [8], or in the context of outcome-control [9]. A recent review appears in [10].\n  \n  * (C24) page 24, first paragraph, \"Through counterfactual reasoning we can explore fairness...\": there is a huge amount of complexity hidden here, and the term counterfactual reasoning is used very cursory. A recent organization of various quantities appearing in the literature can be found in [11, Fig. 12]. \n  \n  * (C25) Section 6.4.1: quotation marks around ' The term \"counterfactual\" ' need to be fixed.\n  \n  * (C26) Section 6.4.1: A note on counterfactual explanations: this name is also used for various approaches that are not at all causal, for example in the algorithmic recourse literature. This is perhaps worth pointing out / being explicit about.\n  \n  * (C27) Section 6.4.2: Once again, more recent and closely realted references for fairness+RL can probably be found.\n  \n  * (C28) Section 7.1: I personally find that structural learning (inferring the causal diagram) is extremely difficult in practice. I have used structural learning in practice many times, and never obtained anything that is remotely useful in practice. I wonder if it is worth discussing some of the limitations. Structural learning for Semi-Markovian models is very involved. Also, in this context, partially oriented graphs [12] could be discussed/mentioned as a future direction. \n\n   * (C29) Paragraphs 7.5 and 7.6 are quite interesting discussions.\n  \n  \n### References\n\n[1] Bareinboim, Elias, et al. \"On Pearl’s hierarchy and the foundations of causal inference.\" Probabilistic and causal inference: the works of Judea Pearl. 2022. 507-556.\n\n[2] Bareinboim, Elias, Andrew Forney, and Judea Pearl. \"Bandits with unobserved confounders: A causal approach.\" Advances in Neural Information Processing Systems 28 (2015).\n\n[3] Pearl, Judea, and Elias Bareinboim. \"External validity: From do-calculus to transportability across populations.\" Probabilistic and causal inference: The works of Judea Pearl. 2022. 451-482.\n\n[4] Shpitser, Ilya, and Judea Pearl. \"What counterfactuals can be tested.\" arXiv preprint arXiv:1206.5294 (2012).\n\n[5] Lee, Sanghack, Juan D. Correa, and Elias Bareinboim. \"General identifiability with arbitrary surrogate experiments.\" Uncertainty in artificial intelligence. PMLR, 2020.\n\n[6] Correa, Juan, Sanghack Lee, and Elias Bareinboim. \"Nested counterfactual identification from arbitrary surrogate experiments.\" Advances in Neural Information Processing Systems 34 (2021): 6856-6867.\n\n[7] Bareinboim, Elias, Jin Tian, and Judea Pearl. \"Recovering from selection bias in causal and statistical inference.\" Probabilistic and Causal Inference: The Works of Judea Pearl. 2022. 433-450.\n\n[8] Huang, Wen, Lu Zhang, and Xintao Wu. \"Achieving counterfactual fairness for causal bandit.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 6. 2022.\n\n[9] Plecko, Drago, and Elias Bareinboim. \"Causal Fairness for Outcome Control.\" arXiv preprint arXiv:2306.05066 (2023).\n\n[10] Carey, Alycia N., and Xintao Wu. \"The causal fairness field guide: Perspectives from social and formal sciences.\" Frontiers in Big Data 5 (2022): 892837.\n\n[11] Plecko, Drago, and Elias Bareinboim. \"Causal fairness analysis.\" arXiv preprint arXiv:2207.11385 (2022).\n\n[12] Perkovic, Emilija. \"Identifying causal effects in maximally oriented partially directed acyclic graphs.\" Conference on Uncertainty in Artificial Intelligence. PMLR, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Overall, I read with interest the survey paper on causal reinforcement learning. In my assessment, the paper does a pretty good job at covering various aspects of the field. The paper is well-written, informed, and it is clear that a substantial amount of effort has gone into it.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See strengths and weaknesses.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GrrSnQsbb8", "reviewer": "Reviewer_nERm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. There are not many similar survey papers that investigate the combination of causality and RL at the submission date of this manuscript. This work should have high empirical value to inspire future RL works.\n2. The example of the cause of scurvy is intuitive. Using this example throughout the entire paper makes many concepts easy to understand.\n3. The comparison and categorization of existing works is a good summarization. This will be helpful to future works that seek to find comparable baselines.\n\n**Weaknesses:**\n1. More intuitive examples about causal RL. The cause of the scurvy example still belongs to the causality field as there is no sequential decision-making problem. I think more concrete examples about when causality appears in RL tasks and how causality influences the decisions could make this paper attractive to broader readers. Maybe add one figure with several tasks and explanations. \n2. Section 2.1 is too long (a “brief” introduction with 3.5 pages). As this survey targets an advanced topic of the intersection between causality and RL, it should assume the readers have basic background knowledge. The authors can refer the readers to other references, which I believe there is a lot in both RL and causality fields. Another reason I think the background of causality should be short is that Causal RL mainly focuses on how to use causality tools in RL, so it would be better to move some contents to later sections and explain causality concepts under RL contexts.\n3. The specific advantages of Causal RL should be mentioned in the abstract to highlight the main contribution. I think the most valuable part of this paper is the following three sections:\n    * Enhancing Sample Efficiency through Causal Reinforcement Learning\n    * Advancing Generalizability and Knowledge Transfer through Causal Reinforcement Learning.\n    * Addressing Spurious Correlations through Causal Reinforcement Learning\n\n   These sections greatly summarize three advantages of using causality in RL, so maybe the authors should consider briefly mentioning them in the abstract.\n\n4. The title of section 6 is a little confusing. The connection between this section and previous sections is also not clear. This section itself is easy to understand but the title and structure may need to be revised. For example, a similar title to previous sections: “Addressing Explainability, Fairness, and Safety through Causal RL”.\n5. Lastly, I have a clarification question about the difference between spurious correlation and generalizability. As discussed in this paper [1], spurious correlation can be treated as one type of distribution shift and can be solved with the distributional robust framework. One example is that the compositional generalization can be caused by an unobserved confounder, i.e., when the value of the confounder is different, the environment gives different types of combinations of tasks. Does the author think these two advantages can be merged together?\n\n[1] Wenhao Ding, Laixi Shi, Yuejie Chi, and Ding Zhao. \"Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation.\" arXiv preprint arXiv:2307.07907 (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper investigates the intersection of two important and promising areas, reinforcement learning and causality. The motivation of this combination is that reinforcement learning agents lack a fundamental understanding of the world and must therefore learn from scratch through numerous trial-and-error interactions, which could probably be fixed by causality that formalizes knowledge in a systematic manner and leverages invariance for effective knowledge transfer. Particularly, the contribution can be summarized as below:\n* explores the improvements of Causal RL over traditional RL methods.\n* identifies key challenges in RL that can be effectively addressed by causality.\n* provides a categorization of existing Causal RL methods.\n* summarizes major unresolved issues and promising research directions in causal RL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see points 1-4 of the weaknesses.", "broader_impact_concerns": "There is no ethical concern as far as I know.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provided comprehensive list of related work on the causal reinforcement learning (Reviewer 6Lf7, 64gv), and promising research directions (64gv). The current version of the submission has been improved based on previous comments (H8ZM), and reviewers remaining concerns  have been addressed by the authors (nERm). 

Although there is still a concern about the framing and categorization of the causal RL, as a survey, I agree with most of the reviewers that there is no solid reason for rejection. 

I suggest to accept the paper as is.
Decision: Accept (poster)