Causal Influence Detection for Improving Efficiency in Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ) is a promising route towards versatile and dexterous artificial agents . Learning from interactions can lead to robust control strategies that can cope with all the intricacies of the real world that are hard to engineer correctly . Still , many relevant tasks such as object manipulation pose significant challenges for RL . Although impressive results have been achieved using simulation-to-real transfer [ 1 ] or heavy physical parallelization [ 2 ] , training requires countless hours of interaction . Improving sample efficiency is thus a key concern in RL . In this paper , we approach this issue from a causal inference perspective . When is an agent in control of its environment ? An agent can only influence the environment by its actions . This seemingly trivial observation has the underappreciated aspect that the causal influence of actions is situation dependent . Consider the simple scenario of a robotic arm in front of an object on a table . Clearly , the object can only be moved when contact between the robot and object is made . Generally , there are situations where immediate causal influence is possible , while in others , none is . In this work , we formalize this situation-dependent nature of control and show how it can be exploited to improve the sample efficiency of RL agents . To this end , we derive a measure that captures the causal influence of actions on the environment and devise a practical method to compute it . Knowing when the agent has control over an object of interest is important both from a learning and an exploration perspective . The learning algorithm should pay particular attention to these situations because ( i ) the robot is initially rarely in control of the object of interest , making training inefficient , ( ii ) physical contacts are hard to model , thus require more effort to learn and ( iii ) these states are enabling manipulation towards further goals . But for learning to take place , the algorithm first needs data that contains these relevant states . Thus , the agent has to take its causal influence into account already during exploration . We propose several ways in which our measure of causal influence can be integrated into RL algorithms to address both the exploration , and the learning side . For exploration , agents can be rewarded with a bonus for visiting states of causal influence . We show that such a bonus leads the agent to quickly discover useful behavior even in the absence of task-specific rewards . Moreover , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . our approach allows to explicitly guide the exploration to favor actions with higher predicted causal impact . This works well as an alternative to -greedy exploration , as we demonstrate . Finally , for learning , we propose an off-policy prioritization scheme and show that it reliably improves data efficiency . Each of our investigations is backed by empirical evaluations in robotic manipulation environments and demonstrates a clear improvement of the state-of-the-art with the same generic influence measure . 2 Related Work . The idea underlying our work is that an agent can only sometimes influence its surroundings . This rests on two basic assumptions about the causal structure of the world . The first is that the world consists of independent entities , in accordance with the principle of independent causal mechanisms ( ICM ) [ 3 ] , stating that the world ’ s generative process consists of autonomous modules . The second assumption is that the potential influence that entities have over other entities is localized spatially and occurs sparsely in time . We can see this as explaining the sparse mechanism shift hypothesis , which states that naturally occurring distribution shifts will be due to local mechanism changes [ 4 ] . This is usually traced back to the ICM principle , i.e . that interventions on one mechanism will not affect other mechanisms [ 5 ] . But we argue that it is also due to the limited interventional range of agents ( or , more generally , physical processes ) , which restricts the breadth and frequency of mechanism-changes in the real world . Previous work has used sparseness to learn disentangled representations [ 6 , 7 ] , causal models [ 8 ] , or modular architectures [ 9 ] . In the present work , we show that taking the localized and sparse nature of influence into account can also strongly improve RL algorithms . Detecting causal influence , informally , means deciding whether changing a causal variable would have an impact on another variable . This involves causal discovery , that is , finding the existence of arrows in the causal graph [ 10 ] . While the task of causal discovery is unidentifiable in general [ 11 ] , there are assumptions which permit discovery [ 12 ] , in particular in the time series setting we are concerned with [ 13 ] . Even if the existence of an arrow is established , the problem remains of quantifying its causal impact , for which various measures such as transfer entropy or information flow have been proposed [ 14–18 ] . We compare how our work relates to these measures in Sec . 4.1 . The intersection of RL and causality has been the subject of recent research [ 19–23 ] . Close to ours is the work of Pitis et al . [ 24 ] , who also use influence detection , albeit to create counterfactual data that augments the training of RL agents . In Sec . 5 , we find that our approach to action influence detection performs better than their heuristic approach . Additionally , we demonstrate that influence detection can also be used to help agents explore better . To this end , we use influence as a type of intrinsic motivation . For exploration , various signals have been proposed , e.g . model surprise [ 25– 27 ] , learning progress [ 27 , 28 ] , empowerment [ 29 , 30 ] , information gain [ 31–33 ] , or predictive information [ 34 , 35 ] . Inspired by causality , Sontakke et al . [ 36 ] introduce an exploration signal that leads agents to experiment with the environment to discover causal factors of variation . In concurrent work , Zhao et al . [ 37 ] propose to use mutual information between the agent and the environment state for exploration . As in our work , the agent is considered a separate entity from the environment . However , their approach does not discriminate between individual situations the agent is in . Causal influence is also related to the concept of contingency awareness from psychology [ 38 ] , that is , the knowledge that one ’ s actions can affect the environment . On Atari games , exploring through the lens of contingency awareness has led to state-of-the-art results [ 39 , 40 ] . 3 Background . We are concerned with a Markov decision process 〈S , A , P , r , γ〉 consisting of state and action space , transition distribution , reward function and discount factor.1 Most real world environments consist of entities that behave mostly independently of each other . We model this by assuming a known state space factorization S = S1 × . . .× SN , where each Si corresponds to the state of an entity . 1We use capital letters ( e.g . X ) to denote random variables , small letters to denote samples drawn from particular distributions ( e.g . x ∼ PX ) , and caligraphy letters to denote graphs , sets and sample spaces ( e.g . x ∈ X ) . We denote distributions with P and their densitites with p . 3.1 Causal Graphical Models . We can model the one-step transition dynamics at time step t using a causal graphical model ( CGM ) [ 3 , 10 ] over the set of random variables V = { S1 , . . . , SN , A , S′1 , . . . , S′N } , consisting of a directed graph G ( see Fig . 1a ) and a conditional distribution P ( Vi | Pa ( Vi ) ) for each node Vi ∈ V , where Pa ( Vi ) is the set of parents of Vi in the causal graph . We assume that the joint distribution PV is Markovian with respect to the graph [ 3 , Def . 6.21 ( iii ) ] , that is , its density exists and factorizes as p ( v1 , . . . , v|V| ) = |V|∏ i=1 p ( vi | Pa ( Vi ) ) . ( 1 ) In a CGM , we can model a ( stochastic ) intervention do ( Vi : = q ( vi | Pa ( Vi ) ) ) on variable Vi by replacing its conditional p ( vi | Pa ( Vi ) ) in Eq . 1 with the distribution q ( vi | Pa ( Vi ) ) [ 3 ] . Here , Vi could e.g . be a state component Si , or the agent ’ s action A . Thus , whereas a probabilistic graphical model represents a single distribution , a CGM represents a set of distributions [ 4 ] . The causal graph that we assume is shown in Fig . 1a . Within a time step , there are no edges , i.e . no instantaneous effects , except for the action which is computed by the policy π ( A | S ) . Between time steps , the graph is fully connected . The reason is that whenever an interaction between two components Si and Sj , however unlikely , is possible , it is necessary to include an arrow Si → S′j ( and vice versa ) . Nevertheless , during most concrete time steps , there should be no interaction between entities , reflecting the assumption that the state components represent independent entities in the world . In particular , the agent ’ s “ sphere of influence ” ( depicted in blue in Figs . 1b and 1c ) is limited – its action A can only sparsely affect other entities . Thus , in this paper , we are interested in inferring the influence the action has in a specific state configuration S = s , that is , the local causal model in s. Definition 1 . Given a CGM with distribution PV and graph G , we define the local CGM induced by observing X = x with X ⊂ V to be the CGM with joint distribution PV|X=x and the graph GX=x resulting from removing edges from G until PV|X=x is causally minimal with respect to the graph . Causal minimality tells us that each edge X → Y in the graph must be “ active ” , in the sense that Y is conditionally dependent on X given all other parents of Y [ 3 , Prop . 6.36 ] . 3.2 The Cause of an Effect . When is an agent ’ s action A = a the cause of an outcome B = b ? Answering this question precisely is surprisingly non-trivial and is studied under the name of actual causation [ 10 , 41 ] . Humans would answer by contrasting the actual outcome to some normative world in which A = a did not happen , i.e . they would ask the counterfactual question “ What would have happened normally to B without A = a ? ” [ 41 ] . Algorithmitizing this approach poses certain problems . First , it requires a “ normal ” outcome which can be difficult to compute as it depends on the behavior of the different actors in the world . Second , it requires to actually observe the world ’ s state without the agents interference . Such a “ no influence ” action may not be available for every agent . Instead , we are inspired by an alternative approach , the so-called “ but-for ” test : “ B = b would not have happened but for A = a. ” In other words , A = a was a necessary condition for B = b to occur , and under a different value for A , B would have had a different value as well . This matches well with an algorithmic view on causation : A is a cause of B if the value of A is required to determine the value of B [ 42 ] . The but-for test yields potentially counterintuitive assessments . Consider a robotic arm close to an object but performing an action that moves it away from the object . Then this action is considered a cause for the position of the object in this step , as an alternative action touching the object would have led to a different outcome . Algorithmically , knowing the action is required to determine what happens to the object – all actions are considered to be a cause in this situation . Importantly , this implies that we can not differentiate whether individual actions are causes or not , but can only identify whether or not the agent has causal influence on other entities in the current state .

Summary: The paper introduces a state-dependent measure of “causal action influence” (CAI) to detect to what extent the agent can affect the next state. The high-level conceptual contribution (as stated by the authors at the beginning of Section 2) is that the measure should in fact be state-dependent. After introducing the measure and a tractable approximation for it, the authors demonstrate various ways in which it can be used to improve the efficiency of learning: 1. Guiding exploration by rewarding the agent for visiting states where it is determined to have more influence 2. Guiding exploration by selecting actions which are determined to have the most influence (among those generated by the policy) 3. A prioritized experience replay sampling scheme which upweights trajectories where the agent visited more states where it has high influence These approaches are also complementary, and combining the three leads to better results than any alone.