Causal Influence Detection for Improving Efficiency in Reinforcement Learning
Authors: Maximilian Seitzer
Bernhard Schölkopf
Georg Martius
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
causal inference
exploration
intrinsic motivation
prioritized replay

Abstract: Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.

Causal Influence Detection
for Improving Efficiency in Reinforcement Learning
Maximilian Seitzer
Bernhard Schölkopf MPI for Intelligent Systems Tübingen, Germany ba@tue.. cefttue.mpg.c.. mpg.de
Georg Martius
MPI for Intelligent Systems Tübingen, Germany
MPI for Intelligent Systems
Tübingen, Germany
maximilian, sei weiitzeettu.nnng www.uee..pp............ mpg.deg...............
georg www.imsttuuuueeeeem wwwpaatteeepp....e.mmm mpg.
Abstract
Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off- off-poolcy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.
Introduction
Reinforcement learning (RL) is a promising route towards versatile and dexterous artificial agents. Learning from interactions can lead robust control strategies that can cope with all the intricacies of the real world that are hard to engineer correctly. Still, many relevant tasks such as object manipulation pose significant challenges for RI. Although impressive results have been achieved using simulation-to-reat transfer [1] or heavy physical parallelization
[2], training requires countless hours of interaction. Improving sample efficiency is thus a key concern in RL. this paper, we approach this issue from causal inference perspective.
When is an agent in control of its environment? An agent can only influence the environment by its actions. This seemingly trivial observation has the underappreciated aspect that the causal influence of actions is situation dependent. Consider the simple scenario of a robotic arm in front of an object on a table. Clearly, the object can only be moved when contact between the robot and object is made. Generally, there are situations where immediate causal influence possible, while others, none is. In this work, we formalize this situation-depender nature of control and show how it can be exploited to improve the sample efficiency RI. agents. To this end, we derive measure that captures the causal influence of actions on the environment and devise practical method to compute it.
Knowing when the agent has control over an object of interest important both from a learning and an exploration perspective. The learning algorithm should pay particular attention to these situations because (i) the robot initially rarely in control of the object of interest, making training inefficient, (ii) physical contacts are hard to model, thus require more effort to learn and (iii) these states are enabling manipulation towards further goals. But for learning to take place, the algorithm first needs data that contains these relevant states, Thus, the agent has to take its causal influence into account already during exploration.
We propose several ways in which our measure of causal influence can be integrated into RL algorithms to address both the exploration, and the learning side. For exploration, agents can be rewarded with a bonus for visiting states of causal influence. We show that such bonus leads the agent quickly discover useful behavior even in the absence of task-specific rewards. Moreover,
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

our approach allows to explicitly guide the exploration to favor actions with higher predicted causal impact. This works well as an alternative to é-greedy exploration, as we demonstrate. Finally, for learning, we propose an off-policy prioritization scheme and show that it reliably improves data efficiency. Each of our investigations is backed by empirical evaluations in robotic manipulation environments and demonstrates clear improvement of the state-of-the-an with the same generic influence measure.
2
Related Work
The idea underlying our work is that an agent can only sometimes influence its surroundings. This rests on two basic assumptions about the causal structure of the world. The first is that the world consists of independent entities, in accordance with the principle of independent causal mechanisms (ICM) [3], stating that the world's generative process consists of autonomous modules. The second assumption is that the potential influence that entities have over other entities is localized spatially and occurs sparsely in time. We can see this as explaining the sparse mechanism shift hypothesis, which states that naturally occurring distribution shifts will be due to local mechanism changes [4]. This is usually traced back to the ICM principle, i.e. that interventions on one mechanism will not affect other mechanisms [5]. But argue that it is also due to the limited interventional range of agents (or, more generally, physical processes), which restricts the breadth and frequency of mechanism-changes in the real world. Previous work has used sparseness to learn disentangled representations [6, 7], causal models [8], or modular architectures [9]. In the present work, we show that taking the localized and sparse nature of influence into account can also strongly improve RL algorithms.
Detecting causal influence, informally, means deciding whether changing a causal variable would have an impact on another variable. This involves causal discovery, that is, finding the existence of arrows in the causal graph [10]. While the task of causal discovery is unidentifiable
general [11], there are assumptions which permit discovery [12], in particular in the time series setting are concerned with [13]. Even the existence of an arrow is established, the problem remains of quantifying its causal impact, for which various measures such transfer entropy information flow have been proposed [14-18]. We compare how our work relates to these measures in Sec. 4.1.
The intersection of RI. and causality has been the subject of recent research [19-23]. Close to ours is the work of Pitis et al. [24], who also use influence detection, albeit to create counterfactual data that augments the training of RL agents. In Sec. 5, we find that our approach to action influence detection performs better than their heuristic approach. Additionally, we demonstrate that influence detection can also be used to help agents explore better. To this end, intrinsic motivation. For exploration, various signals have been proposed, e.g. model surprise [25- 27]. learning progress [27, 28], empowerment [29, 30], information gain [31-33], or predictive information [34, 35]. Inspired by causality, Sontakke et al. [36] introduce an exploration signal that leads agents to experiment with the environment discover causal factors of variation. In concurrent work, Zhao et al. [37] propose to use mutual information between the agent and the environment state for exploration. As in our work, the agent is considered separate entity from the environment. However, their approach does not discriminate between individual situations the agent is in. Causal influence is also related to the concept of contingency awareness from psychology [38], that is, the knowledge that one's 's actions can affect the environment. On Atari games, exploring through the lens of contingency awareness has led state-of-the of-the-rrt results [39, 40].
influence
type of
3 Background
We are concerned with a Markov decision process (S, A, P.r, consisting of state and action space, transition distribution, reward function and discount factor.' Most real world environments consist of entities that behave mostly independently of each other. We model this by assuming a known state space factorization S S₁ ... x SN. where each St corresponds to the state of an entity.
We use capital letters (e.g. X) to denote random variables, small letters to denote samples drawn from particular distributions (c.g. x ~ Px), and caligraphy letters to denote graphs, sets and sample spaces (e.g. E X²). We denote distributions with 1 and their densitites with p.

---

t+1
S
P(S)
S
S
A
S2
2
S/2
(b) No influenec of A on S/
S =
8
S
SN
S'N
x(4 | 99)
(a) Causal Graph g
(c) Influence of A on Si and S
Figure 1: Causal graphical model capturing the environment transition from state S' by action A, factorized into state components. (a): Viewed globally over all time steps, all components of the state and the action can influence all state components the next time step. (b, c): Given a situation S = s, some influences may or may not not hold in the local causal graph 95=.. In this paper. our aim is to detect which influence the action has on S", i.e. the presence of the red arrows.
3.1 Causal Graphical Models
We can model the one-step transition dynamics at time step / using causal gruphical model (CGM) [3, 10] over the set of random variables V .., SN A, S Si,..,, S/ consisting of directed graph G (see Fig. Ia) and conditional distribution P(V, Pa(V()) for each node V2 E V, where Pa (V)) is the set of parents of Vú in the causal graph. We assume that the joint distribution Pv is Markovian with respect to the graph [3, Def. 6.21 (iii), that is, its density exists and factorizes as
[V]
p(
) = II p(vs Pa(V))). 1=1
In a CGM, we can model a (stochastic) intervention do(V; q(t; Pa(V:))) on variable V; by replacing its conditional p(,; Pa(V))) in Eq. with the distribution q(u; Pa(V))) [3]. Here, Vi could e.g. be state component S, or the agent's action A. Thus, whereas a probabilistic graphical model represents a single distribution, a CGM represents a set distributions [4].
The causal graph that we assume is shown in Fig. 1a. Within a time step, there are no edges, i.e. no instantancous effects, except for the action which computed by the policy x S). Between time steps, the graph is fully connected. The reason is that whenever an interaction between two components S, and Sy. however unlikely, is possible, it is necessary to include an arrow S, - Sy (and vice versa). Nevertheless, during most concrete time steps, there should be no interaction between entities, reflecting the assumption that the state components represent independent entities in the world. In particular, the agent's 's "sphere of influence" (depicted in blue in Figs. 1b and Ic) is limited its action A can only sparsely affect other entities. Thus, in this paper, we are interested in inferring the influence the action has in specific state configuration S = s, that is, the local causal model in
Definition 1. Given a CGM with distribution Py and graph G. we define the local CGM induced by observing X z with X C V to be the CGM with joint distribution Pyjx== and the graph Gx=x resulting from removing edges from G until Pyrx= is causally minimal with respect to the graph.
Causal minimality tells us that each edge + Y the graph must be "active", the sense that Y is conditionally dependent on X given all other parents of Y [3, Prop. 6.36).
3.2 The Cause of an Effect
When is an agent's 's action A the cause of an outcome B b? Answering this question precisely is surprisingly non-trivial and is studied under the name of actual causation [10, 41]. Humans would answer by contrasting the actual outcome to some normative world in which A = a did not happen, i.e. they would ask the counterfactual question "What would have happened normally to B without A = a?" [41]. Algorithmitizing this approach poses certain problems. First, requíres "normal" outcome which can be difficult to compute as it depends on the behavior of the different actors in
3

---

the world. Second, it requires to actually observe the world's 's state without the agents interference. Such a "no influence" action may not be available for every agent. Instead, we are inspired by an alternative approach, the so-called "but-for" test: "B = b would not have happened but for A = a." In other words, = a was a necessary condition for = b to occur, and under a different value for A, B would have had different value as well. This matches well with an algorithmic view on causation: A is a cause of B if the value of Á is required to determine the value of [42].
The but-for test yields potentially counterintuitive assessments. Consider a robotic arm close to an object but performing an action that moves it away from the object. Then this action is considered a cause for the position of the object in this step, as an alternative action touching the object would have led a different outcome. Algorithmically, knowing the action is required to determine what happens to the object- - all actions are considered to be cause in this situation. Importantly, this implies that we cannot differentiate whether individual actions are causes or not, but can only identify whether or not the agent has causal influence on other entities in the current state.
4 Causal Influence Detection
As the previous discussion showed, having causal influence is dependent on the situation the agent in, rather than the chosen actions. We characterize this as the agent being in control, analogous to similar notions in control theory [43]. Formally, using the causal model introduced in Sec. 3, we define the agent to be in control of S in state S if there is an edge A + S¹J in the local causal graph Gs=s under all interventions do(A A := x (als)) with n having full support. The following proposition states when such an edge exists (proofs in Suppl. A.1).
Let GS=s be the graph of the local CGM induced by 5 = 8. There is an edge A + S// in Gs= under the intervention do A == [a)ss) if and only if 5 L | =s
Proposition 1.
To detect when the agent is in control, we can intervene with a policy. The following proposition gives conditions under which conclusions drawn from one policy generalize many policies.
Proposition 2. If there is intervention do(A:= <(als)) under which S # A S s, this depen- dence holds under all interventions with full support, and the agent is in control of S in If there is an intervention do(A: == {als)) with π having full support under which S 1L A S independence holds under all possible interventions and the agent is not in control of S in s.
this
4.1 Measuring Causal Action Influence
Our goal is to find a state-dependent quantity that measures whether the agent is in control of S As Prop. I tells us, control (or its absence) is linked to the independence S IL A S = 8. A well-known measure of dependence is the conditional mutual information (CMI) [44] which is zero for independence. We thus propose to use (pointwise) CMI as a measure of causal action influence (CAD) that can be thresholded to get a classification of control (see Suppl. A.2 for derivation):
((s)==I(Spp) [()):=(($}AA\ s) = EnNT [ (Ps)le.
Ps71.)
We want this measure to be independent of the particular policy used in the joint distribution P(S, S'). This is because we might not be able to sample from evaluate this policy (e.g. in off-policy RI,, the data stems from a mixture of different policies). Fortunately, Prop. 2 shows that to detect control, it is sufficient to demonstrate (in- (nn)ddependence for a single policy with full support. Thus, we can choose a uniform distribution over the action space as the policy: . (AA := U(A).
Let us discuss how CAI relates to previously suggested measures of (causal) influence. Transfer entropy [14] is a non-linear extension of Granger causality [45] quantifying causal influence in time series under certain conditions. CAI is similar to one-step, local transfer entropy [17] with the difference that CAI conditions on the full state S. Janzing et al. [18] put forward a measure of causal strength fulfilling several natural criteria that other measures, including transfer entropy, fail to satisfy. Suppl. A.3, we show that CAI is pointwise version of Janzing et al. causal strength, for policies not conditional on the state S (adding further justification for the choice of uniform random policy). Furthermore, we can relate CAI to notions of controllability [43]. Decomposing (s) as HSS) () HSS | A, s), where H denotes the conditional entropy [44], we can interpret CAI

---

as quantifiying the degree to which S can be controlled in s, accounting for the system's 's intrinsic uncertainty that cannot be reduced by the action.
In the context of RI., empowerment [29, 30, 46] is a well-known quantity used for intrinsically- motivated exploration that leads agents to states of maximal influence over the environment. Em- powerment, for a state s, is defined as the channel capacity between action and a future state, which coincides with max, C(s) for one-step empowerment. CAI can thus be seen as a non-trivial lower bound of empowerment that is easier to compute. However, CAI differs from empowerment in that it does not treat the state space as monolithic and specific to an entity. In Sec. 6.1, we demonstrate that an RL agent maximizing CAI quickly achieves control over its environment.
4.2 Learning to Detect Control
Estimating CMI is hard problem many levels: it involves computing high dimensional integrals, representing complicated distributions and having access to limited data; strictly speaking, each conditioning point is seen only once in continuous spaces. In practice, one thus has resort to an approximation. Non-parametric estimators based on nearest neighbors [47, 48] or kernels methods [49] are known not scale well to higher dimensions [50]. Instead, approach the problem by learning neural network models with suitable simplifying assumptions.
Expanding the KL divergence in Eq. 2, can write CAI
p(s 5,a)
= s s) = Es s.a log
(3)
p
1 T r(a)da
To compute this term, we estimate the transition distribution p(s' | s, a) from data. We then approximate the outer expectation and the transition marginal p(s') 8) by sampling K actions from the policy æ. This gives us the estimator
(i(s)
= K
1 1 Σ
DKL
p(s)) s, ())
1 K
E plss s, a())
with (a a (K) lid S.. Here, we replaced the infinite mixture p(s)) |s) with finite mixture, p(s () 1 ΣE p(s; |s, (.). and used Monte-Carke approximate the expectation. Poole et al. [51] show that this estimator is a lower bound converging to the true mutual information A| A|S =) K increases (assuming, however, the true density p(s', s.a)).
K
To compute the KL divergence itself, we make the simplifying assumption that the transition dis- tribution p(5; s,a) is normally distributed given the action, which is reasonable in the robotics environment we are targeting. This allows us to estimate the KI. without expensive MC sampling by using an approximation for mixtures of Gaussians from Durrieu et al. [52]. We detail the exact formula we use in Suppl. A.4.
With the normality assumption, the density itself can be learned using a probabilistic neural network and simple maximum likelihood estimation. That is, we parametrize p(s') s, a,a) as (iss Po(s, a), (s,ss,)), where μa, of are the outputs of a neural network fo(s, a). We find the parameters 0 by minimizing the negative log-likelihood over samples D {{s(0), (+), ( collected by some policy (the univariate case shown here also extends to the multivariate case):
N
(()) 1o (s!1), (s!(),a(1)) ()))
0" = arg min
+ log (a))
(5)
N
203 a())
There are some intricacies regarding the policy that collects the data for model training and the sampling policy π that is used to compute CAL First of all, the two policies need to have overlapping support to avoid evaluating the model under actions never seen during training. Furthermore, the data policy is different from the sampling policy the model is biased to some degree. This suggests to use π for collecting the data; however, as we use a random policy, this will not result in interesting data in most environments. The bias can be reduced by sampling actions from и during data collection with some probability and only train on those. In practice, however, we find to obtain better performing models by training all data despite potentially being biased.
5

---

IDSLIDE
(b) PETCHIPCCANNPPACE
(c) PETCHIPCCKANDPLCCH
1.0
1.0
True Positive Rate
0.8
True Positive Rate
0.8
Average Precision
0.8
0.6
0.4
0.4
CAT
CAT Entropy Attention
0.4
CAT
0.2
Entropy Attention
Entropy Attention
0.0
0.2
0.4 False Positive Rate
0.0
0.2
0.4
0.6
0.8
1.0
0%
5%
10%
15%
20%
False Positive Rate
Noisc Level
Figure 2: Causal influence detection performance. (a, b) ROC curves on IDSLIDE and FETCH- PICKANDPI.AGE environments. (c) Average precision for FETCHPICK added state noise. Noise level is given as percentage of one standard deviation over the dataset.
depending
Table 1: Results for evaluating causal influence detection on different environments. We measure area under the ROC curve (AUC), average precision (AP), and the best achievable F-score (F)).
IDSLIDE
FETCHPICKANDPLACE
AUC
AP
Fi
AUC
AP
CAI (ours)
1.00 + 0.00
0.98 0.00
0.95 = 0.01
0.97 0.01
0.96 + 0.00
0.89 0.00
Entropy
0.96 0.00
0.47
0.01
0.50 = 0.01
0.84 0.00
0.73 1 0.00
0.78 0.00
Attention [24]
0.42 + 0.31
0.13 + 0.14
0.18 = 0.17
0.46 + 0.06
0.44 + 0.04
0.62 - 0.00
Contacts
0.89
0.78
0.88
0.79
0.77
0.73
5 Empirical Evaluation of Causal Influence Detection
In this section, we evaluate the quality of our proposed causal influence detection approach in relevant environments. As simple test case, we designed an environment (IDSLIDE) in which the agent must slide an object to a goal location by colliding with it. Furthermore, we test on the FETCHPICK ANDPLACC environment from OpenAl Gym [53], its original setting and when adding Gaussian noise to the observations to simulate more real-world conditions. In both environments, the target variables of interest are the coordinates of the object. Note that we need the true causal graph at derive this information from the simulation. For
each time step for the evaluation. For IDSLIDE, the pick and place environment with its non-trivial dynamics, we resort to a heuristic of the possible movement range of the robotic arm in one step. Detailed information about the setup is provided in Suppls. B and
For our method, we use CAI estimated according to Eq. 4 (with K = 64) as classification score that thresholded to gain binary decision. We compare with a recently proposed method [24] that uses the attention weights of Transformer model [54] to model influence. Moreover, we compare with an Entropy baseline that uses HS' 8) as score and a Contact baseline based on binary contact information from the simulator. We show the test results over 5 random seeds in Table and Fig. 2. We observe that CAI is able reliably detect causal influence and no other baseline is able to do so. When increasing the observation noise, the performance drops gracefully for CAI shown in Fig. 2c. Suppl. contains more experimental results, including visualization CAl's behavior.
Improving Efficiency in Reinforcement Learning
Having established the efficacy of our causal action influence (CAI) measure, develop several approaches to use to improve RL algorithms. We will empirically verify the following claims in robotic manipulation environments: CAI improves sample efficiency and performance by (i) better state exploration through an exploration bonus, (ii) causal action exploration, and (iii) prioritizing experiences with causal influence during training.
consider the environments FETCHPUSH, FETCHPICKANDPHA from OpenAl Gym [55], and BETCHROTTABLE which is our modification containing rotating table (explained in Suppl. B.3).

---

1.0
4.0
1.0
Fraction of Episode
Average Reward YCAT
Sucpess Rate
10.6
2.4
0.6
0.4
1.6
0.4
Abceus
0.0
0.3
Object Moved
0.1 ... 0.4
Object Held in Air
0.2
0.0
0.0
0.0
0
12
16
20
Rollouts x x1000
Rollouts 1000
Figure 3: Intrinsically motivated learning on PETCHPICCANNPPLCC The reward is only TCAT measured on the object coordinates.
Figure 4: Exploration bonus improves perfor-
mance in SETCIUPICKANDPLAA Sensitivity
the bonus reward scale Abonus.
These environments are goal-conditioned RL tasks with sparse rewards, meaning that each episode, new goal is provided and the agent only receives a distinct reward upon reaching it. We use DDPG [56] with hindsight experience replay (HER) [57] as the base RL algorithm, combination that achieves state-of-the-ar results in these environment. The influence detection model is trained online on the data collected from an RL agent learning to solve its task. Since our measure C requires an entity of interest, we choose the coordinates of the object (as S,). In all experiments, we report the mean success rate with standard deviation over 10 random seeds. More information about the experimental settings can be found in Suppl. F.
6.1 Intrinsic Motivation to Seek Influence
Causal Action Influence as Reward Bonus. We hypothesize that useful for an agent to be intrinsically motivated to gain control over its environment. We test this hypothesis by letting the agent maximize the causal influence it has over entities of interest. This can achieved by using our influence measure as reward signal. The reward signal can be used on its own, motivation-type objective, or in conjunction with a task-specific reward as an exploration bonus. In the former case, expect the agent discover useful behaviors that can help it master task-oriented skills afterwards; in the latter case, we expect learning efficiency to improve, especially in sparse extrinsic reward scenarios. Concretely, for state s, we define the bonus FCAI(s) = Cs(ss, and the total reward as r(s) = Trask (s) + Abonus ICAI(s), where Trask (s) is the task reward, and Abonus is hyperparameter.
intrinsic
Experiment on Intrinsically Motivated Learning. We first test the behavior of the agent in the absence of any task-specific reward on the SATTHHPCCKANPLLACA environment. Interestingly, the agent learned to grasp, lift, and hold the object in the air already after 2000 episodes, as shown in Fig. 3. The results demonstrate that encouraging causal control over the environment well suited to prepare the agent for further tasks it might have to solve.
Impact of CAI Reward Bonus. Second, we are interested in the impact of adding an exploration bonus. In Fig. 4, we present results on the FETCHPICKANDPLACC environment when varying the reward scale Abomus Naturally, the exploration bonus needs to be selected in the appropriate scale as value too high will make dominate the task reward. If selected correctly, the sample efficiency is improved drastically; for example, we find that the agent reaches success rate of 60% four-times faster than the baseline (DDPG+HER) without any bonus (Ahomus = 0).
6.2 Actively Exploring Actions with Causal Influence
Exploration via bonus rewards favors the re- visitation of already seen states. An alternative approach to exploration uses pro-active planning to choose exploratory actions. In our case, we can make use of our learned influence estimator to pick actions which we expect will have the largest causal effect on the agent's surroundings. From a causal viewpoint, the resulting agent can be seen as experimenter that performs planned interventions in the environment to verify its beliefs. Should the actual outcome differ from the expected outcome, subsequent model updates can integrate the new data self-correct the causal influence estimator.
Following Actions with the Most Causal Influence.
7

---

1.0
1.0
Success Rate
Success Rate
0.6
0.6
Bonus
0.4
0.4
VIME
Active Exploration
Fineebbe Dis.
0%
75%
CAI-Act.
25%
100%
CAI-Bonus
50%
CAI-Act. Bonnus
0.0
0.0
0
12 16
20
0
8
12
16
20
Rollouts 1000
Rollouts x1000
Figure Performance of active exploration in PETCHIPICKANDLLCCT depending on the fraction of exploratory actions chosen actively (Eq. 6) from total of 30% exploratory actions.
Figure 6: Experiment comparing exploration strategies on DETCCpPCCAANDLU bination of active exploration and reward bonus yields the largest sample efficiency.
The com-
Concretely, given the agent being in state 8, we choose the action that has the largest contribution to the empirical mean in Eq. 4:
arg max ag (all) o(k))
DKL
p( 8,
K
Σ p(s/ 8, a(())
(6)
with {a( ..... a (K) iid deviation from the expected outcome under all actions. For states s where the the agent is not in control, i.e. (s) = 0, the action selection is uniform at random.
πT Intuitively, the selected action will be the one which results in maximal
Active Exploration in Practice. Can active exploration replace c-greedy exploration? To gain insights, study the impact of the fraction of actively chosen exploration actions. For every exploratory action (e is 30% in our experiments), we choose an action according to Eq. 6 the specified fraction of the time, and otherwise random action. Figure 5 shows that any amount of active exploration improves over simple random exploration. Active causal action exploration can improve the sample efficiency roughly by a factor of two.
Combined CAI Exploration. We also present the combination of reward bonus and active ex- ploration and compare our method with VIME, another exploration scheme based on information- theoretic measures [33]. In contrast to our method, VIME maximizes the information gain about the state transition dynamics. Further, compare ensemble disagreement [58], which in effect minimizes epistemic uncertainty about the transition dynamics. We compare different variants of VIME and ensemble disagreement in Suppl. D, and display only their best versions here. Figure quantifies the superiority of all CAI variants (with ensemble disagreement as a viable alternative) and shows that combining the two exploration strategies compounds increase sample efficiency further. In the figure, CAI uses 100% active exploration and Abonus 0.2 as the bonus reward scale.
6.3 Causal Influence- based Experience Replay
Prioritizing According to Causal Influence. We will now propose another method using CAL, namely to inform the choice samples replayed to the agent during off-policy training. Typically, past states sampled uniformly for learning. Intuitively, focusing those states where the agent has control over the object of interest (as measured by CAI) should improve the sample efficiency. We implement this idea using prioritization scheme that samples past episodes in which the agent had more influence more frequently. Concretely, we define the probability p(i) of sampling any state from episode i (of episodes) in the replay buffer as
p(0)
M (M + - rank Σ {())))) t=1
(())
with
(
(7,8)
ΣM Di=1 p(t)
T
where T' is the episode length, and p(() is the priority of episode i. The priority of an episode i is based on the (inverse) rank of the episode (rank) when sorting all M episodes according to their total influence (i.e. of state influences). We call this causal action influence prioritization (CAI-P).

---

FETCHPUSH
FETCHPICCANDPLACE
HETCHROTTABLE
1.0
1.0
1.0
CAT-P
Success Rate
0.8
Sucoess Rate
0.8
Sucpess Rate
0.8
EBP
PER
0.6
0.6
0.6
HER
CAI-P
CAI-P EBP
0.4
EBP
0.4
0.4
PER
PER
0.2
0.2
0.2
HER
HER
0.0
0.0
0.0
8
10
16
20
8
12
16
20
Rollouts 1000
Rollouts x 1100
Rollouts
~1000
Figure 7: Prioritizing experience replay in different manipulation environments. Comparison of causal action influence prioritization (CAI-P) against baselines: the energy-based method (EBP) [60] with privileged information, prioritized experience replay (PER) [59], and HER without prioritization.
This scheme is similar Prioritized Experience Replay [59], with two differences: instead of using the TD error for prioritization, we use the causal influence measure. Furthermore, instead of prioritizing individual states, we prioritize episodes and sample states uniformly within episodes. This is because the information about the return that can be achieved from an influence state still needs to be propagated back to non-influenced states by TD updates, which requires sampling them.
Infleence-Based Prioritization in Manipulation Tasks, We compare our influence- based prioritization (CAI-P) against no prioritization in hindsight experience replay (HER) (a strong baseline for multi-goal RL), and two other prioritization schemes: prioritized experience replay (PER) [59] and energy- based prioritization (EBP) [60]. Especially EBP is a strong method for the environments we considering as it uses privileged knowledge of the underly- ing physics to replay episodes based on the amount of energy that is transferred from agent the object to manipulate. All prioritization variants are equipped with HER as well. The FETCHROTTAA TABLE environment, shown in Fig. 8, is an interesting test bed as the object can move through the table rotation without the control of the agent. The results, shown in Fig. 7. reveal that causal influence prioritization can speed up learning drastically. Our method is on par or better than the anergy-basee (oracle) method EBP and improves over PER by a factor of 1 1.5-2.5 in learning speed (at 60% success rate). Finally, in Suppl. D, we combine all our proposed improvements and show that FRECHPIICANNPPLCCT can be solved up to 95% success rate in just 3000 episodes.
Figure 8: FETCH ROTTABLE. The table rotates peri- odically.
Discussion
In this work, we show how iituatiin-dependder causal influence detection can help improve rein- forcement learning agents. To this end, we derive a measure of local causal action influence (CAI) and introduce a data-driven approach based neural network models to estimate it. We showcase using CAI as an exploration bonus, as a way to perform active action exploration, and to prioritize in experience replay. Each of our applications yields strong improvements in sample efficiency. We expect that there are further ways to use our causal measure in RL, e.g. for credit assignment.
Our work has several limitations. First, assume full observability of the state, which simplifies the causal inference problem as there no confounding between an agent's ss action and its effect. Under partial observability, our approach could still be applicable using latent variable models [61]. Second, we require an available factorization of the state into causal variables. The problem of automatically learning causal variables from digenhimeniio data is open [4] and our method would likely benefit from advances in this field. Third, the accurate estimation of our measure relies on a correct model. We found that deep networks struggle at times pick up the causal relationship between actions and entities. How to design models with appropriate inductive biases for cause-effed inference an open question [3, 62].
An intriguing future direction is to extend our work to influence detection between entities, a prerequisite for identifying multi-step influences of the agent on the environment. Being able to model such indirect interventions would bring us closer to "artificial scientists" agents that can perform planned experiments to reveal the latent causal structure of the world.

---

Acknowledgments and Disclosure of Funding
The authors thank Andrii Zadaianchuk and Dominik Zietlow for many helpful discussions and providing feedback on the text. Furthermore, the authors would also like to thank Sebastian Blaes environment. The authors thank the International Max Planck for creating the FETCHROTTABHE Research School for Intelligent Systems (IMPRS-IS) for supporting Maximilian Seitzer. GM and BS are members of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AT Center (FKZ: (0338)))...
References
[1] Marein Andrychowiez, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[2] Dmitry Kalashnikov, Alex Irpan, Peter Pastor Sampedro, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference Robot Learning 2018, 2018. URL. https:::: //axxiv...//////////a/r// org/pdf 1806. 10293.
[3] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
[4] B. Schölkopf, F. Locatello, S. Bauer, R. Nan Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Towards causal representation learning. Proceedings of the IEEE, 2021. doi: 109910./ 202 2021.3058954.
[5] Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas- Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 4036-4044, PMLR, 10-15 Jul 2018. URL httpp///proceedings mlr. wwwpppressse.ee srtess/v80/paa aata0//aaaacandoolaaaaa.oo html..
[6] Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sebastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to dis entangle causal mechanisms. In 8th International Conference on Learning Representations (ICLR), 2020. URL http::::///ppereeeeeeeeeee wtwprreeeeeeeeeeroueeeeeeeeeeeee
nwt. net//Eorum?dd=xxyllggepp..
[7] Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages (348-6359) PMLR, 13-18 Jul 2020. URL http://///rro mlr press / http////////ee11...aaa.o html.
[8] Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C. Mozer, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. ArXiv, abs/1910.01075. 2019.
[9] A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf. Recurrent independent mechanisms. In 9th International Conference Learning Representations (ICLR), 2021.
[10] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009.
[11] Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT Press, Cambridge, MA, 2nd edition, 2000.
[12] J. Peters, J. Mooij, D. Janzing, and B. Schölkopf. Identifiability of causal graphs using functional models. In G. Cozman and A. Pfeffer, editors, 27th Conference on Uncertainty in Artificial Intelligence, pages 589-598, Corvallis, OR, 201 AUAI Press
10

---

[13] Michael Eichler. Causal Inference in Time Series Analysis. Wiley, 2012. ISBN 978 177811199 119945710. doi: 00000//97811994571000
[14] Thomas Schreiber. Measuring information transfer. Physical Review Letters, 85 2:461-4, 2000.
[15] Nihat Ay and David C. Krakauer. Geometric robustness theory and biological networks. Theory in Biosciences, 125(2):93-121, 2007. ISSN 1431-7613. doi: https://doi.or//10..00 1016/ j. (thbio.2006.06.000 URL https://wwwt www.eeedreeece...cc.........
$1431761306000255.
[16] N. Ay and D. Polani. Information flows in causal networks. Adv. Complex Syst., 11:17-41, 2008.
[17] Joseph T. Lizier. The local information dynamics of distributed computation in complex systems. PhD thesis, University of Sydney, 2013. URL http:://////ntte wtfo/1/24/2111.17777
(18) D. Janzing, D. Balduzzi, M. Grosse- Wentrup, and B. Schölkopf. Quantifying causal influences. Anmals of Statistics, 41(5):2324-2358, 2013. URL httLhtttt:/poojeeeeuuee
org/ euclid.
aos nos/1383661266.
[19] E. Bareinboim, A. Forney, and J. Pearl. Bandits with unobserved confounders: A causal approach. In Advances in Neural Information Processing, volume 28. Curran Associates, Inc., 2015.
[20] Chaochao Lu, B. Schölkopf, and José Miguel Hernández-Lobato. Deconfounding reinforcement learning in observational settings. ArXiv, abs/ 0bs//181..1156.. 2018.
[21] Danilo Jimenez Rezende, Ivo Danihelka, George Papamakarios, N. Ke, Ray Jiang, T. Weber, K. Gregor, Hamza Merzic, Fabio Viola, J. Wang, Jovana Mitrovic, F. Besse, loannis Antonoglou, and Lars Buesing. Causally Correct Partial Models for Reinforcement Learning. ArXiv, abs/2002.02838 2020.
[22] Lars Buesing, T. Weber, Yori Zwols, Sébastien Racanière, A. Guez, J. Lespiau, and N. Heess. Woulda, Coulda, Shoulda: conntterfactallyy-uudee Policy Search. In 7th International Confer- ence on Learning Representations (ICLR), 2019.
[23] J. Zhang and E. Bareinboim. Near-optimal reinforcement learning in dynamic treatment regimes. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2019.
[24] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
[25] Schmidhuber. Curious model-building control systems. Proceedings IEEE International Joint Conference on Neural Networks, pages 1458-1463 vol.2, 1991. doi: 10.1 10.1109//CCNN.190 19911 170605.
[26] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supervised prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 488 489, 2017.
[27] Sehastian Blaes, Marin Vlastelica, Jia-Jie Zhu, and Georg Martius. Control What You Can Intrinsically motivated task-planning agent. In Advances in Neural Information Processing, pages 12520-1220 12531. Curran Associates, Inc., 2019. URL http: http::tttppeeeeee h/ppp//pap///eppeeep nips cc/parrr/ can www--nnttnnnnccllln motllyymmtvtteeedeeoo -tweeattddddad--a-eeeeeeaaa---e-
cal-cootoolwhaate va waw.raattdd-ttt---llannnnaeeeeee wwitiila-annnnnniieee pdf.
agent.
[28] Cédric Colas, Pierre-Y Yves Oudeyer, Olivier Sigaud, Pierre Fournier, and Mohamed Chetouani. CURIOUS: intrinsically motivated modular multi-goal reinforcement learning. In International Conference Machine Learning (ICML' 19), pages 1331-13400 2019.
[29] A.S. Klyubin, D. Polani, and C.I. Nehaniv. Empowerment: a universal agent-centric measure of control. 2005 IEEE Congress on Evolutionary Computation, volume 1, pages 128-135 Vol.1, 2005, doi: 10. LLL.109//EE..2005555547777

---

[30] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Cortes, N. Lawrence, Lee, M. Sugiyama. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: http////oooeeeeneeeeeee cceere/p.cc/pappe/22055 file/ file/ (ile/e0040611
Paper .df pdf.
[31] Jan Storck, Sepp Hochreiter, and Jürgen Schmidhuber.
Reinforcement driven information acquisition in non non-deterministic environments. In ICANN'95, pages 159-164, 1995.
[32] Daniel Little and Friedrich Sommer. Learning and exploration in action-perception loops. Frontiers in Neural Circuits, 7:37, 2013. ISSN 1662-5110. doi: 10.3389ffncir.2113.0000 URL https: / /www. frontiersin org/ wtgg/rrtice//ii httee//11....////0 338.33389/3330 for9/fnncr:.../
[33] Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. cc
neurips. .df.
[34] Georg Martius, Ralf Der, and Nihat Ay. Information driven self-organization of complex robotic behaviors. PLoS ONE, 8(5):e33400, 2013. doi: 10. 10.137//journal.po pone.00033000.. URL http://dx.
www..o......oooooooo..oo...o.... 10. 1371/ wwn.ppon....oooe 00634000.
[35] Keyan Zahedi, Georg Martius, and Nihat Ay. Linear combination of one-step predictive informa- tion with an external reward in episodic policy gradient setting: critical analysis. Frontiers in Psychology, 4(801), 2013. ISSN 1664-1078. doi: 10.3389/fpsy.. 201p:ss.....1..... 2013.000011 3.000. URL http:// www www.forotiersnn.cmmmmmmmm org/ cognitiiiiiieocceeee wwtiveeesseeeeeeeeeeeeeeeeseeeee 10. 11..033.9//50 fesg. 2013. 00801/ abstractt..........
[36] S. Sontakke, A. Mehrjou, L. Itti, and B. Schölkopf. Causal curiosity: RI agents discovering self-supervised experiments for causal representation learning. In Proceedings of 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9848-9858. PMLR, July 2021. URL https: /tttpp:///rrcee sott/kontak html.
http:::/t///:eeeeeeee http:///pocceeega//e mlr, .pless/
[37] Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. International Conference on Learning Representations, 2021. URL https://openrevvexx www./nee//ee.eee httee//frrmmmmdd0th281113313
[38] John S. Watson. The development and generalization of contingency awareness early infancy: Some hypotheses. SterilllPalmee Quarterly of Behavior and Development, 12(2):123-13: 1966. ISSN 00260150. URL http: / /www. hwtp:wwwtttt....ooooo..oo....... jstor. org/ www.aaaeeeeee//.
[39] Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration in reinforcement learning. In International Conference on Learning Representations (ICLR), 2019. URL forumPid-HyxxGBAACYY.
[40] Yuhang Song, Jianyi Wang, Thomas Lukasiewicz, Zhenghua Xu, Shangtong Zhang, Andrzej Wojeicki, and Mai Xu. Mega-Reward: Achieving human-level play without extrinsic re- wards. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))5826-5833, Apr. 2020. doi: 10. 10.1609/aaai, hal.1609aaai.v34404.60000 v34i04.6040. URL https://ojs aaai. arg/index... pwaiddee...ppieeee AAAI/
article/view/ee 6040
[41] Joseph Y. Halper. Actual Causality. The MIT Press, 2016. ISBN 9780262035026.
[42] J. Pearl, M. Glymour, and N. Jewell. Causal Inference in Statistics: A Primer. Wiley, 2016. ISBN 978-1-119-18684-7
[43] Hugo Touchette and Seth Lloyd. Information-theoretio approach to the study control systems. Physica A: Statistical Mechanics and its Applicutions, 331(11:14- 2004. ISSN 0378-4371. doi: scincc//articee/p//////eeee11119
0007. htRls::::: http://///wwssseeeeeeeeeeeee sciencedirect. com/
12

---

[44] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley Series in Telecommunications and Signal Processing. Wiley--nterscience, 2nd ed edition, 2006.
[45] C. W. J. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3))424-338 1969. URL http://www. /ttttor.rg//table//// 1912791.
[46] Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-Ar Introduction, pages 67-114. Springer Berlin Heidelberg, Berlin, Heidelberg, 2014. ISBN 978-3-642-53373433 doi: 10.10077997-33444--577-444 4.
URL https://doi 1
[47] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review. E, Statistical, nonlinear, and soft matter physics, 69:066138, 07 2004, doi: 10.1 00.110//hyysReeE.6..0666
[48] Jakob Runge. Conditional independence testing based nearest-neighhor estimator of condi- tional mutual information. Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-Firy First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 938-947. PMLR, 09-11 Apr 2018. URL http://prroeedinggs mlr. prees//84/// www////gg/8/aaaa. html.
[49] Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, and james m robins. Nonparametric von Mises estimators for entropies, divergences and mutual informations.
C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://prooeedingss wtw.eeddininnnnnnns...eeeeee httpppppe//011//iiiii 061 oat.lc/a.oo2266eeoeeeeeeeeeeeeee
.df.
[50] Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. for Strongly Dependent Variables. Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 Proceedings of Machine Learning Research, pages 277-286, San Diego, California, USA, 12 May 2015. PMLR. URLhttp: http://pooceeennneee/ee/e mlr. pwwps/v3s....p /aa////....... 15. html.
Efficient Estimation of Mutual Information
[51] Ben Poole. Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pages 5171-5180, PMLR, 09-15 Jun 2019. URL http:///proceedinss mlr. www.pprsssssseee http://///poo/e///// pwwp:pool...... html.
[52] Jean-Louis Durrieu, J. Thiran, and Finnian Kelly. Lower and upper bounds for approximation of the kullback-leinn leibler divergence between gaussian mixture models. 2012 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 4833 4836, 2012.
[53] Greg Broekman, Vicki Cheung, Ludwig Pettersson, J. Schneider, John Schulman, Jie Tang, and W. Zaremba. OpenAl Gym. ArXiv, abs/1606.01144 2016.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https://proceede wwwpeediinnnnnuuisso.o. www.nee.../.pppppeeeeee pttpcc////ppe////////e/ www.2295447ee111b0033eee.o....e.
Paper. pdf.
[55] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, J. Schneider, Joshua Tobin, Maciek Chociej. P. Welinder, V. Kumar, and W. Zaremba. Multi- reinforcement learning: Challenging roboties environments and request for research. ArXiv, abs/1802.09464, 2018.
[56] T. Lillicrap, Jonathan J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations (ICLR), 2016.
13

---

[57] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[58] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via dis- agreement. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5062-5071 PMLR, 09-15 Jun 2019. URL https: mlr. www.:mw..........t.. Satess/v9//pathaa11990 html.
/ //proceedings.. mml.rdiinnmmm..........mmm.
[59] Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016.
[60] Rui Zhao and Volker Tresp. Energy- based hindsight experience prioritization. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 113-122. PMLR, 29-31 Oct 2018.
[61] Christos Louizos, Uri Shalit, Joris M Mooij. David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: http:///poccee...cee/e////c wwweeedding..euuiiss.... wwtriiss.ccpappr///200 (aabbadedeBBddffccaaa444aaaa -appee.edd..pr......... pdf.
[62] Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition.
ArXiv, abs/2011.1509) 2020.
[63] John R. Hershey and Peder A. Olsen. Approximating the Kullback Leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing- ICASSP '07, volume 4, pages IV-317-IV-300, 2007. doi: 10. 10.1109/ICASSP 2007.366913.
[64] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages
2012.
[65] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[66] Andrew M. Saxe, James L. McClelland, and S. Ganguli. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations (ICLR), 2014.
[67] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICL.R, 2018
[68] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: pttpp//prppppppppppp.a....pppppp wwweeripps.ccpppp//200550 2t11//772272B6633355555511144442
www.iinnn.nnuussrioomeo Paper pdf.
[69] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
14


Reviews:
[{"review_id": "rAmYcfdpd0I", "reviewer": "Reviewer_FgCT", "summary": "The authors study how to determine what factors in an environment can be controlled by the agent using a causal setup. One of the contributions is a perspective that causal influence is dependent on situation. They propose to use the conditional mutual information to measure if a future state variable is independent of action conditioned on the current state. Of course, estimating this value is difficult. There are some simplifying assumptions made: that the per state variable transition distribution is Gaussian and various quantities can be predicted by neural networks. After an empirical experiment validating that their Causal Influence Detection approach outperforms other attention-based methods, the authors propose 3 ways in which this detector can be used to improve sample efficiency in RL: 1) better state exploration, 2) causal action exploration and 3) prioritized replay with causal influence.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: Causality and RL are a combination that has received a lot of recent interest, but I found the connection to be well-used in this work to justify the objective. While the objective itself is not particularly novel, the authors derive a way to estimate the quantity in a way that seems to work well empirically. \nSome of the three ways in which this quantity is used have certainly been tried before but, again, the combination of the novel measure and the empirical results are compelling.\n\nquality: This paper was a pleasure to read. The experiments are thoughtfully designed to break down the different contributions and show sensitivity to introduced hyper parameters. Baselines are also thoughtfully chosen and explained for most of the experiments. This leads me to some questions and suggestions:\n\nFor Figure 4, why was no other exploration method compared to? There are similar intrinsic reward-based exploration methods I would assume to perform similarly, like a count-based method or one based on “surprise,” as measured by epistemic uncertainty via ensembles, or even just simple entropy regularization.\n\nclarity: Same as above. I found the background thoughtfully presented, derivation of the method well explained, and experiments well-designed.\n\nsignificance: The contributions of this work are substantial. An effective method for computing mutual information quantities in robotics environments will be of great use to RL methods for those applications. The authors have also presented three ways to use that quantity to improve sample efficiency over SOTA in reasonable environments significantly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "mCMXWfcFJO", "reviewer": "Reviewer_HRby", "summary": "This work is built upon the idea of independent causal mechanisms in reinforcement learning environments where an agent only has limited inference over other entities in a given situation. Based on this premise, the authors introduce a measure called situation-dependent causal influence (CAI), which is conditional mutual information. The authors show how to estimate CAI and how to use it in exploration and RL policy training. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the idea behind the work: because an RL agent has only a limited invervetional range, exploration and experience replay in RL could be more focused to be effective. At the same time, there are a couple of issues that need to be addressed/clarified. \n\nFirst, it seems that simple heuristics could achieve the same goal. Specifically, the work seems to be targeting rotis environments (line 192). In this case, my feeling is that CAI identification be rather straightforward; similar to the heuristics used in the simulation (line 216) - the agent can only affect the states of the possible movement range of the robotic arm in one step. Clearly, exploration should be more in these areas. Would that suffice? When would we need the more sophisticated algorithm proposed in Sec. 4?  \n\nSecond, in the current evaluation, the authors evaluate how effective their algorithms can estimate CAI in Sec. 5. Clearly, their algorithm outperforms others significantly. There are two questions. First, how about a simple heuristic, like the one mentioned above? Second, just estimating CAI itself is not the ultimate goal. The ultimate goal is training an RL agent effectively. In Figure 5, no other exploration method is compared except fixed \\epsilon-greedy. In Figure 6, only VIME and its variants are compared. In the literature of effective exploration, many algorithms have been proposed.   A more thorough evaluation is needed to really demonstrate the benefit of the proposed algorithm. As shown in Figure 7, CAI has similar performance with EBP in two cases and outperforms EBP in one. \n\nThere are a few strong assumptions needed in the paper. On one hand, I understand the necessity of starting with something simple. On the other hand, I wonder how much it would be possible to relax some of these assumptions. First, the model requires a known space factorization. How critical is this assumption? In robotics environments, could this be done relatively easily using some heuristics? Second, the paper is based on the causality framework, which I greatly appreciate. However, causality has its strong assumptions. I was wondering if the work really depends on the causality framework. It seems to me that the key component is to evaluate the conditional mutual information, presented in Eq. 2. Could we start from that? That would make the work much more general. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion is satisfactory.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "c-y9QZ-2w_l", "reviewer": "Reviewer_VNL6", "summary": "Summary. In reinforcement learning problems, many entities often act independently and only have influence on a small set of other entities. The paper aims to use causality to discover which components of state space are affected by agent’s action and leverage this fact for a more efficient reinforcement learning. To detect cause influence between state component and action, the paper uses mutual information between that component and action (conditioned on current state). Finally, it leverages the detected causal influence to (i) help with exploration by using it as exploration bonus, (ii) take actions with maximum causal influence and (iii) prioritizing experiences with causal influence for training policy. Overall, the paper shows improvement over baseline algorithms in state based robotic tasks.\n \nMy main concern is the applicability of the method to image-based settings. I do think latent variable models will cause issues and I am not sure about the extent of the problem. To be fair, authors mention this issue as well in the discussion section. It would be interesting to see how this method performs if state variables are replaced by latent variables (coming from some pretrained generative model; authors can use some expert data for pretraining the generative model for a start to get good features)  \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper presents a novel way to incorporate detection of causal influence in state-based RL framework.\n\nQuality: The claims made in the paper are backed up by proper experiments.\n\nClarity: The paper is clearly written.\n\nSignificance: The paper does a good job of showing importance of leveraging causality for sample efficient RL. However, it’s hard to know the long term significance without knowing the applicability of the method to image based setting.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is not addressed in the paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hr"}, "extra_text": {}}, {"review_id": "bB8bHCysx5d", "reviewer": "Reviewer_BGJy", "summary": "The paper introduces a state-dependent measure of “causal action influence” (CAI) to detect to what extent the agent can affect the next state. The high-level conceptual contribution (as stated by the authors at the beginning of Section 2) is that the measure should in fact be state-dependent. After introducing the measure and a tractable approximation for it, the authors demonstrate various ways in which it can be used to improve the efficiency of learning:\n1. Guiding exploration by rewarding the agent for visiting states where it is determined to have more influence\n2. Guiding exploration by selecting actions which are determined to have the most influence (among those generated by the policy)\n3. A prioritized experience replay sampling scheme which upweights trajectories where the agent visited more states where it has high influence\nThese approaches are also complementary, and combining the three leads to better results than any alone.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the approach of estimating causal influence in the environment. It does seem intuitive that most “real-world” RL tasks would involve causal relationships due to their sequential nature, and uncovering those relationships should lead to more efficient learning.\n\nThe CAI estimator presented in the paper has several complementary uses in efficient exploration and learning, so the contribution of the work is clear. The primary downside of the approach is that it requires some domain knowledge, such as a known factorization of the state space and full observability, as well as an idea about which coordinate(s) are likely to contain causal relationships which are useful for exploration. However, in the cases where such information is known, there is a clear benefit to using it.\n\nOne suggestion to make the paper even stronger would be to evaluate against another baseline or two for the exploration experiments, e.g. a model-based surprise or random network distillation. The paper mentions a variety of approaches in the related work, but only compares against VIME.\n\nThe paper is clear and well-written. I have no significant issues with the writing or organization. The appendix contains useful information such as hyperparameter values and additional plots. I particularly like Figure S4, which plots the CAI over time along with corresponding images of the robot.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I would like to congratulate the authors on an excellent investigation. This paper makes very nice progress in the combination of RL and causal inference, and all the reviewers and myself found the work exciting and important. 

Please make sure to address the reviewers' feedback in the final version of the paper.
Decision: Accept (Poster)