Causal Influence Detection for Improving Efficiency in Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ) is a promising route towards versatile and dexterous artificial agents . Learning from interactions can lead to robust control strategies that can cope with all the intricacies of the real world that are hard to engineer correctly . Still , many relevant tasks such as object manipulation pose significant challenges for RL . Although impressive results have been achieved using simulation-to-real transfer [ 1 ] or heavy physical parallelization [ 2 ] , training requires countless hours of interaction . Improving sample efficiency is thus a key concern in RL . In this paper , we approach this issue from a causal inference perspective . When is an agent in control of its environment ? An agent can only influence the environment by its actions . This seemingly trivial observation has the underappreciated aspect that the causal influence of actions is situation dependent . Consider the simple scenario of a robotic arm in front of an object on a table . Clearly , the object can only be moved when contact between the robot and object is made . Generally , there are situations where immediate causal influence is possible , while in others , none is . In this work , we formalize this situation-dependent nature of control and show how it can be exploited to improve the sample efficiency of RL agents . To this end , we derive a measure that captures the causal influence of actions on the environment and devise a practical method to compute it . Knowing when the agent has control over an object of interest is important both from a learning and an exploration perspective . The learning algorithm should pay particular attention to these situations because ( i ) the robot is initially rarely in control of the object of interest , making training inefficient , ( ii ) physical contacts are hard to model , thus require more effort to learn and ( iii ) these states are enabling manipulation towards further goals . But for learning to take place , the algorithm first needs data that contains these relevant states . Thus , the agent has to take its causal influence into account already during exploration . We propose several ways in which our measure of causal influence can be integrated into RL algorithms to address both the exploration , and the learning side . For exploration , agents can be rewarded with a bonus for visiting states of causal influence . We show that such a bonus leads the agent to quickly discover useful behavior even in the absence of task-specific rewards . Moreover , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . our approach allows to explicitly guide the exploration to favor actions with higher predicted causal impact . This works well as an alternative to -greedy exploration , as we demonstrate . Finally , for learning , we propose an off-policy prioritization scheme and show that it reliably improves data efficiency . Each of our investigations is backed by empirical evaluations in robotic manipulation environments and demonstrates a clear improvement of the state-of-the-art with the same generic influence measure . 2 Related Work . The idea underlying our work is that an agent can only sometimes influence its surroundings . This rests on two basic assumptions about the causal structure of the world . The first is that the world consists of independent entities , in accordance with the principle of independent causal mechanisms ( ICM ) [ 3 ] , stating that the world ’ s generative process consists of autonomous modules . The second assumption is that the potential influence that entities have over other entities is localized spatially and occurs sparsely in time . We can see this as explaining the sparse mechanism shift hypothesis , which states that naturally occurring distribution shifts will be due to local mechanism changes [ 4 ] . This is usually traced back to the ICM principle , i.e . that interventions on one mechanism will not affect other mechanisms [ 5 ] . But we argue that it is also due to the limited interventional range of agents ( or , more generally , physical processes ) , which restricts the breadth and frequency of mechanism-changes in the real world . Previous work has used sparseness to learn disentangled representations [ 6 , 7 ] , causal models [ 8 ] , or modular architectures [ 9 ] . In the present work , we show that taking the localized and sparse nature of influence into account can also strongly improve RL algorithms . Detecting causal influence , informally , means deciding whether changing a causal variable would have an impact on another variable . This involves causal discovery , that is , finding the existence of arrows in the causal graph [ 10 ] . While the task of causal discovery is unidentifiable in general [ 11 ] , there are assumptions which permit discovery [ 12 ] , in particular in the time series setting we are concerned with [ 13 ] . Even if the existence of an arrow is established , the problem remains of quantifying its causal impact , for which various measures such as transfer entropy or information flow have been proposed [ 14–18 ] . We compare how our work relates to these measures in Sec . 4.1 . The intersection of RL and causality has been the subject of recent research [ 19–23 ] . Close to ours is the work of Pitis et al . [ 24 ] , who also use influence detection , albeit to create counterfactual data that augments the training of RL agents . In Sec . 5 , we find that our approach to action influence detection performs better than their heuristic approach . Additionally , we demonstrate that influence detection can also be used to help agents explore better . To this end , we use influence as a type of intrinsic motivation . For exploration , various signals have been proposed , e.g . model surprise [ 25– 27 ] , learning progress [ 27 , 28 ] , empowerment [ 29 , 30 ] , information gain [ 31–33 ] , or predictive information [ 34 , 35 ] . Inspired by causality , Sontakke et al . [ 36 ] introduce an exploration signal that leads agents to experiment with the environment to discover causal factors of variation . In concurrent work , Zhao et al . [ 37 ] propose to use mutual information between the agent and the environment state for exploration . As in our work , the agent is considered a separate entity from the environment . However , their approach does not discriminate between individual situations the agent is in . Causal influence is also related to the concept of contingency awareness from psychology [ 38 ] , that is , the knowledge that one ’ s actions can affect the environment . On Atari games , exploring through the lens of contingency awareness has led to state-of-the-art results [ 39 , 40 ] . 3 Background . We are concerned with a Markov decision process 〈S , A , P , r , γ〉 consisting of state and action space , transition distribution , reward function and discount factor.1 Most real world environments consist of entities that behave mostly independently of each other . We model this by assuming a known state space factorization S = S1 × . . .× SN , where each Si corresponds to the state of an entity . 1We use capital letters ( e.g . X ) to denote random variables , small letters to denote samples drawn from particular distributions ( e.g . x ∼ PX ) , and caligraphy letters to denote graphs , sets and sample spaces ( e.g . x ∈ X ) . We denote distributions with P and their densitites with p . 3.1 Causal Graphical Models . We can model the one-step transition dynamics at time step t using a causal graphical model ( CGM ) [ 3 , 10 ] over the set of random variables V = { S1 , . . . , SN , A , S′1 , . . . , S′N } , consisting of a directed graph G ( see Fig . 1a ) and a conditional distribution P ( Vi | Pa ( Vi ) ) for each node Vi ∈ V , where Pa ( Vi ) is the set of parents of Vi in the causal graph . We assume that the joint distribution PV is Markovian with respect to the graph [ 3 , Def . 6.21 ( iii ) ] , that is , its density exists and factorizes as p ( v1 , . . . , v|V| ) = |V|∏ i=1 p ( vi | Pa ( Vi ) ) . ( 1 ) In a CGM , we can model a ( stochastic ) intervention do ( Vi : = q ( vi | Pa ( Vi ) ) ) on variable Vi by replacing its conditional p ( vi | Pa ( Vi ) ) in Eq . 1 with the distribution q ( vi | Pa ( Vi ) ) [ 3 ] . Here , Vi could e.g . be a state component Si , or the agent ’ s action A . Thus , whereas a probabilistic graphical model represents a single distribution , a CGM represents a set of distributions [ 4 ] . The causal graph that we assume is shown in Fig . 1a . Within a time step , there are no edges , i.e . no instantaneous effects , except for the action which is computed by the policy π ( A | S ) . Between time steps , the graph is fully connected . The reason is that whenever an interaction between two components Si and Sj , however unlikely , is possible , it is necessary to include an arrow Si → S′j ( and vice versa ) . Nevertheless , during most concrete time steps , there should be no interaction between entities , reflecting the assumption that the state components represent independent entities in the world . In particular , the agent ’ s “ sphere of influence ” ( depicted in blue in Figs . 1b and 1c ) is limited – its action A can only sparsely affect other entities . Thus , in this paper , we are interested in inferring the influence the action has in a specific state configuration S = s , that is , the local causal model in s. Definition 1 . Given a CGM with distribution PV and graph G , we define the local CGM induced by observing X = x with X ⊂ V to be the CGM with joint distribution PV|X=x and the graph GX=x resulting from removing edges from G until PV|X=x is causally minimal with respect to the graph . Causal minimality tells us that each edge X → Y in the graph must be “ active ” , in the sense that Y is conditionally dependent on X given all other parents of Y [ 3 , Prop . 6.36 ] . 3.2 The Cause of an Effect . When is an agent ’ s action A = a the cause of an outcome B = b ? Answering this question precisely is surprisingly non-trivial and is studied under the name of actual causation [ 10 , 41 ] . Humans would answer by contrasting the actual outcome to some normative world in which A = a did not happen , i.e . they would ask the counterfactual question “ What would have happened normally to B without A = a ? ” [ 41 ] . Algorithmitizing this approach poses certain problems . First , it requires a “ normal ” outcome which can be difficult to compute as it depends on the behavior of the different actors in the world . Second , it requires to actually observe the world ’ s state without the agents interference . Such a “ no influence ” action may not be available for every agent . Instead , we are inspired by an alternative approach , the so-called “ but-for ” test : “ B = b would not have happened but for A = a. ” In other words , A = a was a necessary condition for B = b to occur , and under a different value for A , B would have had a different value as well . This matches well with an algorithmic view on causation : A is a cause of B if the value of A is required to determine the value of B [ 42 ] . The but-for test yields potentially counterintuitive assessments . Consider a robotic arm close to an object but performing an action that moves it away from the object . Then this action is considered a cause for the position of the object in this step , as an alternative action touching the object would have led to a different outcome . Algorithmically , knowing the action is required to determine what happens to the object – all actions are considered to be a cause in this situation . Importantly , this implies that we can not differentiate whether individual actions are causes or not , but can only identify whether or not the agent has causal influence on other entities in the current state .

Summary: This work is built upon the idea of independent causal mechanisms in reinforcement learning environments where an agent only has limited inference over other entities in a given situation. Based on this premise, the authors introduce a measure called situation-dependent causal influence (CAI), which is conditional mutual information. The authors show how to estimate CAI and how to use it in exploration and RL policy training.