Intrinsic Social Motivation via Causal Influence in Multi-Agent RL

1 INTRODUCTION . Deep reinforcement learning ( RL ) has made impressive progress on specific tasks with well-defined reward functions , but is still difficult to learn intelligent behavior that generalizes across multiple domains . Intrinsic motivation is a technique for solving this problem by developing general reward functions that encourage an agent to learn across a variety of tasks ( Singh et al. , 2004 ) . Previous approaches to intrinsic motivation have broadly fallen into two categories : ( 1 ) curiosity , or a drive for novelty ( e.g . Pathak et al . ( 2017 ) ; Schmidhuber ( 2010 ) ) , and ( 2 ) empowerment , or a drive to be able to manipulate the environment ( Klyubin et al. , 2005 ) . We posit that this body of work has largely overlooked an important intrinsic motivation that is key to human learning : social interaction . Humans have remarkable social learning abilities ; some authors suggest that it is social learning that has given rise to cultural evolution , and allowed us to achieve unprecedented progress and coordination on a massive scale ( van Schaik & Burkart , 2011 ; Herrmann et al. , 2007 ) . Others emphasize that our impressive capacity to learn from others far surpasses that of other animals , apes , and even other proto-human species ( Henrich , 2015 ; Harari , 2014 ; Laland , 2017 ) . Therefore , we propose an intrinsic reward function designed for multi-agent RL ( MARL ) , which awards agents for having a causal influence on other agents ’ actions . Causal influence is assessed using counterfactual reasoning ; at each timestep , an agent simulates alternate , counterfactual actions that it could have taken , and assesses their effect on another agent ’ s behavior . Actions that lead to relatively higher change in the other agent are considered to be highly influential and are rewarded . We show how this reward is related to maximizing the mutual information between agents ’ actions , and is thus a form of social empowerment . We hypothesize that rewarding influence may therefore encourage cooperation between agents . We also take inspiration from experiments in human cognition , showing that newborn infants are sensitive to correspondences between their own actions and the actions of other people , and use this to coordinate their behavior with others ( Tomasello , 2009 ; Melis & Semmann , 2010 ) . To study our proposed social influence reward in the MARL setting , we adopt the Sequential Social Dilemmas ( SSDs ) of Leibo et al . ( 2017 ) . These are challenging MA environments with a game-theoretic reward structure , similar to Prisoner ’ s Dilemma . For each individual agent , ‘ defecting ’ ( non-cooperative behavior ) has the highest payoff . However , the collective reward will be better if all agents choose to cooperate . The paradoxical payoff structure of these tasks make achieving cooperative social dynamics extremely challenging for typical RL agents . We show that social influence allows agents to learn to cooperate in these environments , and make the following contributions : • We demonstrate that deep RL agents trained with the proposed social influence reward cooperate to attain higher collective reward than baseline deep RL agents ( Mnih et al. , 2016 ) . In some cases , this cooperation is attained because influencer agents learn to use their actions as an emergent communication protocol , analogous to behavior seen in animals ( von Frisch , 1969 ) . • Motivated by the previous point , we apply the influence reward to training deep RL agents to use an explicit communication channel , as in Foerster et al . ( 2016 ) . We demonstrate that the communication protocols trained with the influence reward meaningfully relate to agents ’ actions , and that once again , agents trained with the influence reward achieve better collective outcomes . • We demonstrate that there is a significant correlation between being influenced through communication messages and obtaining higher individual return , suggesting that influential communication is beneficial to the agents that receive it . • Finally , rather than computing social influence using a centralised training framework as in prior work ( e.g . Foerster et al . ( 2017 ; 2016 ) ) , we extend the approach by attaching an internal Model of Other Agents ( MOA ) network to each agent and training it to predict the actions of every other agent . The agent can then simulate counterfactual actions and use its own internal MOA to predict how these will affect other agents , thus computing its own intrinsic influence reward . Using a MOA to predict and reward influence allows us to compute an intrinsic social reward by observing other agents ’ past actions , without a centralised controller , and without requiring access to another agent ’ s reward function . We believe this is an important innovation over prior work ( e.g . ( Hughes et al. , 2018 ; Foerster et al. , 2017 ; 2016 ) ) . When we consider likely future applications of MARL , such as autonomous driving , it becomes apparent that centralised training or the sharing of reward functions are unrealistic assumptions , since autonomous vehicles are likely to be produced by a wide variety of organizations and institutions with mixed motivations . Rather , a social reward function which only depends on observing the behavior of agents acting in the environment , and which can give rise to coordinated , cooperative behavior , represents a more promising approach . 2 METHODS . We consider a MARL Markov game defined by the tuple 〈S , T , A , r〉 , in which multiple agents which do not share weights are trained to independently maximize their own individual reward . The environment state is given by s∈S . At each timestep t , each agent k chooses an action akt ∈A . The actions of allN agents are combined to form a joint action at= [ a0t , ... a N t ] , which produces a transition in the environment T ( st+1|at , st ) , according to the state transition function T . Each agent then receives its own reward rk ( at , st ) , which may depend on the actions of other agents . A history of these variables over time is termed a trajectory , τ = { st , at , rt } Tt=0 . We consider a partially observable setting in which each agent k can only view a portion of the true state , skt . Each agent seeks to maximize its own total expected future reward , Rk= ∑∞ i=0γ irkt+i , where γ is a discount factor . A distributed asynchronous advantage actor-critic approach ( A3C ) ( Mnih et al. , 2016 ) is used to train each agent ’ s independent policy πk . The policy is learned via REINFORCE with baseline ( Williams , 1992 ) . Architecturally , our agents consist of a convolutional layer , fully connected layers , a Long Short Term Memory ( LSTM ) network ( Gers et al. , 1999 ) , and linear layers which output πk and the value function V πk ( s ) . We will refer to the internal LSTM state of agent k at timestep t as ukt . 2.1 INTRINSIC SOCIAL MOTIVATION VIA CAUSAL INFLUENCE . Social influence intrinsic motivation modifies an agent ’ s reward function so that it becomes Rk=αEk+βIk , whereEk is the extrinsic or environmental reward , and Ik is the causal influence reward . We compute Ik by generating counterfactual actions that the agent could have taken at each timestep , and assessing how taking these would have affected other agents ’ behavior . A counterfactual is the estimated probability that “ Y would be y hadX been x , in situationZ=z ” , whereX , Y , andZ are random variables , and x , y and z are their values ( Pearl et al. , 2016 ) . Importantly , it is a counterfactual because we condition on a set of evidence z , and because the assignment X =x is counter to what we actually observed ; in reality , X took on some other value . Figure 1 : Causal diagram of agent A ’ s effect on B ’ s action . We condition on each agent ’ s view of the environment and LSTM state u ( shaded nodes ) , and intervene on aAt ( blue ) . To see how we can compute the causal effect of one agent on another , suppose there are two agents , A and B , and that agent B receives A ’ s action at time t , aAt , as input 1 . Agent B then uses this to compute a distribution over its own action , p ( aBt |aAt , sBt , uBt ) . Because we have built the model for agent B , we know all of its inputs : aAt , s B t , and its own internal LSTM state , u B t , as shown in Figure 1 . This allows us to exactly isolate the causal effect ofA ’ s action onB by conditioning on the values we observed for the other inputs at this timestep ( note that by conditioning on these variables , including the LSTM state uBt , we remove any dependency on previous timesteps in the trajectory ) . We can then intervene on aAt by replacing it with a counterfactual action , do ( ã A t ) . This counterfactual action is used to compute a new estimate of p ( aBt |do ( ãAt ) , sBt , uBt ) . Essentially , the agent asks a retrospective question : “ How wouldB ’ s action change if I had acted differently in this situation ? ” . To simplify notation , let zt=〈uBt , sBt 〉 , so that conditioning on zt is equivalent to conditioning on all relevant background variables ( all shaded variables in Figure 1 ) . We can also forego the do operator , noting that p ( aBt |do ( ãAt ) , zt ) ≡p ( aBt |ãAt , zt ) in this case , because z satisfies the back-door criterion ( Pearl & Mackenzie , 2018 ) . Now , consider averaging over several counterfactuals ãAt . This gives us the marginal policy ofB , p ( aBt |zt ) = ∑ ãAt p ( aBt |ãAt , zt ) p ( ãAt |zt ) —in other words , B ’ s policy ifA were not considered . The discrepancy between the marginal policy ofB and the conditional policy ofB givenA ’ s action is a measure of the causal influence of A on B ; it gives the degree to which B changes its planned action distribution because of A ’ s behavior . Thus , the causal influence intrinsic reward for agentA is IAt =DKL [ p ( aBt |aAt , zt ) ∥∥∥∑ ãAt p ( aBt |zt , ãAt ) p ( ãAt |zt ) ] =DKL [ p ( aBt |aAt , zt ) ∥∥∥p ( aBt |zt ) ] . ( 1 ) 2.2 RELATIONSHIP TO MUTUAL INFORMATION AND EMPOWERMENT . The causal influence reward in Eq . 1 is related to the mutual information ( MI ) between the actions of agents A andB , which is given by I ( AB ; AA|z ) = ∑ aA , aB p ( aB , aA|z ) log p ( a B , aA|z ) p ( aB|z ) p ( aA|z ) = ∑ aA p ( aA|z ) DKL [ p ( aB|aA , z ) ∥∥∥p ( aB|z ) ] , ( 2 ) where we see that theDKL factor in Eq . 2 is the causal influence reward given in Eq . 1 . The connection to mutual information is interesting , because a frequently used intrinsic motivation for single agent RL is empowerment , which rewards the agent for having high mutual information between its actions and the future state of the environment ( e.g . Klyubin et al . ( 2005 ) ; Capdepuy et al . ( 2007 ) ) . To the extent that the social influence reward defined in Eq . 1 is an approximation of the MI , A is rewarded for having empowerment overB′s actions . By sampling N independent trajectories τn from the environment , where A ’ s actions aAn are drawn according to p ( aA|z ) , we perform a Monte-Carlo approximation of the MI ( see e.g . Strouse et al . ( 2018 ) ) , I ( AA ; AB|z ) =Eτ [ DKL [ p ( AB|AA , z ) ∥∥p ( AB|z ) ] ∣∣∣z ] ≈ 1 N ∑ n DKL [ p ( AB|aAn , z ) ∥∥p ( AB|z ) ] . ( 3 ) Thus , in expectation , the social influence reward is the MI between agents ’ actions . Whether the policy trained with Eq . 1 actually learns to approximate the MI depends on the learning dynamics . We calculate the intrinsic social influence reward using Eq . 1 , because unlike Eq . 2 , which gives an estimate of the symmetric bandwidth between A and B , Eq . 1 gives the directed causal effect 1Note that this requires that agentA choose its action beforeB , and thereforeA can influenceB butB can not influence A ; in other words , we must impose a sequential ordering on agents ’ actions , and there can not be mutual influence . We improve upon this approach in Section 2.4 . For now , we allow only a fixed number of agents ( ∈ [ 1 , N−1 ] ) to be influencers , and the rest are influencees . Only an influencer gets the causal influence reward , and only an influencee can be influenced . At each timestep , the influencers choose their actions first , and these actions are then given as input to the influencees . If agent A and B are influencers , and C is an influencee , then C receives both aAt and aBt as input . When computing the causal influence of A on C , we also add aBt to the conditioning set . of the specific action taken by agentA , aAt . We believe this will result in an easier reward to learn , since it allows for better credit assignment ; agent A can more easily learn which of its actions lead to high influence . We also experiment with replacing the KL-divergence with several other measures , including the Jensen-Shannon Divergence ( JSD ) , and find that the influence reward is robust to the choice of measure .

Summary: The paper introduces a new intrinsic reward for MARL, representing the causal influence of an agent’s action on another agent counterfactually. The authors show this causal influence reward is related to maximising the mutual information between the agents’ actions. The behaviour of agents using this reward is tested in a set of social dilemmas, where it leads to increased cooperation and communication protocols, especially if given an explicit communication channel. As opposed to related work, the authors also equip the agents with an internal Model of Other Agents that predicts the actions of other agents and simulates counterfactuals. This allows the method to run in a decentralized fashion and without access to other agents’ reward functions.