Influence-Based Multi-Agent Exploration

1 INTRODUCTION . Reinforcement learning algorithms aim to learn a policy that maximizes the accumulative reward from an environment . Many advances of deep reinforcement learning rely on a dense shaped reward function , such as distance to the goal ( Mirowski et al. , 2016 ; Wu et al. , 2018 ) , scores in games ( Mnih et al. , 2015 ) or expert-designed rewards ( Wu & Tian , 2016 ; OpenAI , 2018 ) , but they tend to struggle in many real-world scenarios with sparse rewards ( Burda et al. , 2019 ) . Therefore , many recent works propose to introduce additional intrinsic incentives to boost exploration , including pseudocounts ( Bellemare et al. , 2016 ; Tang et al. , 2017 ; Ostrovski et al. , 2017 ) , model-learning improvements ( Burda et al. , 2019 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) , and information gain ( Florensa et al. , 2017 ; Gupta et al. , 2018 ; Hyoungseok Kim , 2019 ) . These works result in significant progress in many challenging tasks such as Montezuma Revenge ( Burda et al. , 2018 ) , robotic manipulation ( Pathak et al. , 2018 ; Riedmiller et al. , 2018 ) , and Super Mario games ( Burda et al. , 2019 ; Pathak et al. , 2017 ) . Notably , most of the existing breakthroughs on sparse-reward environments have been focusing on single-agent scenarios and leave the exploration problem largely unstudied for multi-agent settings – it is common in real-world applications that multiple agents are required to solve a task in a coordinated fashion ( Cao et al. , 2012 ; Nowé et al. , 2012 ; Zhang & Lesser , 2011 ) . This problem has recently attracted attention and several exploration strategies have been proposed for transition-independent cooperative multi-agent settings ( Dimakopoulou & Van Roy , 2018 ; Dimakopoulou et al. , 2018 ; Bargiacchi et al. , 2018 ; Iqbal & Sha , 2019b ) . Nevertheless , how to explore effectively in more general scenarios with complex reward and transition dependency among cooperative agents remains an open research problem . ∗Equal Contribution . This paper aims to take a step towards this goal . Our basic idea is to coordinate agents ’ exploration by taking into account their interactions during their learning processes . Configurations where interaction happens ( interaction points ) lie at critical junctions in the state-action space , through these critical configurations can transit to potentially important under-explored regions . To exploit this idea , we propose exploration strategies where agents start with decentralized exploration driven by their individual curiosity , and are also encouraged to visit interaction points to influence the exploration processes of other agents and help them get more extrinsic and intrinsic rewards . Based on how to quantify influence among agents , we propose two exploration methods . Exploration via information-theoretic influence ( EITI ) uses mutual information ( MI ) to capture the interdependence between the transition dynamics of agents . Exploration via decision-theoretic influence ( EDTI ) goes further and uses a novel measure called value of interaction ( VoI ) to disentangle the effect of one agent ’ s state-action pair on the expected ( intrinsic ) value of other agents . By optimizing MI or VoI as a regularizer to the value function , agents are encouraged to explore state-action pairs where they can exert influences on other agents for learning sophisticated multi-agent cooperation strategies . To efficiently optimize MI and VoI , we propose augmented policy gradient formulations so that the gradients can be estimated purely from trajectories . The resulting update rule draws a connection between coordinated exploration and the distribution of individual intrinsic rewards among team members , which further explains why our methods are able to facilitate multi-agent exploration . We demonstrate the effectiveness of our methods on a variety of sparse-reward cooperative multiagent tasks . Empirical results show that both EITI and EDTI allow for the discovery of influential states and EDTI further filter out interactions that have no effects on the performance . Our results also imply that these influential states are implicitly discovered as subgoals in search space that guide and coordinate exploration . The video of experiments is available at https : //sites . google.com/view/influence-based-mae/ . 2 SETTINGS . In our work , we consider a fully cooperative multi-agent task that can be modelled by a factored multi-agent MDP G = 〈N , S , A , T , r , h , n〉 , where N ≡ { 1 , 2 , ... , n } is the finite set of agents , S ≡ ×i∈NSi is the finite set of joint states and Si is the state set of agent i . At each timestep , each agent selects an action ai ∈ Ai at state s , forming a joint action a ∈ A ≡ ×i∈NAi , resulting in a shared extrinsic reward r ( s , a ) for each agent and the next state s′ according to the transition function T ( s′|s , a ) . The objective of the task is that each agent learns a policy πi ( ai|si ) , jointly maximizing team performance . The joint policy π=〈π1 , . . . , πn〉 induces an action-value function , Qext , π ( s , a ) = Eτ [ ∑h t=0 r t|s0=s , a0=a , π ] , and a value function V ext , π ( s ) =maxaQext , π ( s , a ) , where τ is the episode trajectory and h is the horizon . We adopt a centralized training and decentralized execution paradigm , which has been widely used in multi-agent deep reinforcement learning ( Foerster et al. , 2016 ; Lowe et al. , 2017 ; Foerster et al. , 2018 ; Rashid et al. , 2018 ) . During training , agents are granted access to the states , actions , ( intrinsic ) rewards , and value functions of other agents , while decentralized execution only requires individual states . 3 INFLUENCE-BASED COORDINATED MULTI-AGENT EXPLORATION . Efficient exploration is critical for reinforcement learning , particularly in sparse-reward tasks . Intrinsic motivation ( Oudeyer & Kaplan , 2009 ) is a crucial mechanism for behaviour learning since it provides the driver of exploration . Therefore , to trade off exploration and exploitation , it is common for an RL agent to maximize an objective of the expected extrinsic reward augmented by the expected intrinsic reward . Curiosity is one of the extensively-studied intrinsic rewards to encourage an agent to explore according to its uncertainty about the environment , which can be measured by model prediction error ( Burda et al. , 2019 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) or state visitation count ( Bellemare et al. , 2016 ; Tang et al. , 2017 ; Ostrovski et al. , 2017 ) . While such an intrinsic motivation as curiosity drives effective individual exploration , it is often not sufficient enough for learning in collaborative multi-agent settings , because it does not take into account agent interactions . To encourage interactions , we propose an influence value aims to quantify one agent ’ s influence on the exploration processes of other agents . Maximizing this value will encourage agents to visit interaction points more often through which the agent team can reach configurations that are rarely visited by decentralized exploration . In next sections , we will provide two ways to formulate the influence value with such properties , leading to two exploration strategies . Thus , for each agent i , our overall optimization objective is : Jθi [ πi|π−i , p0 ] ≡ V ext , π ( s0 ) + V int , π i ( s0 ) + β · I π −i|i , ( 1 ) where p0 ( s0 ) is the initial state distribution , π−i is the joint policy excluding that of agent i , and V int , πi ( s ) is the intrinsic value function of agent i , I π −i|i is the influence value , β > 0 is a weighting term . In this paper , we use the following notations : r̃i ( s , a ) = r ( s , a ) + ui ( si , ai ) , ( 2 ) V πi ( s ) = V ext , π ( s ) + V int , πi ( s ) , ( 3 ) Qπi ( s , a ) = r̃i ( s , a ) + ∑ s′ T ( s′|s , a ) V πi ( s′ ) , ( 4 ) where ui ( si , ai ) is a curiosity-derived intrinsic reward , r̃i ( s , a ) is a sum of intrinsic and extrinsic rewards , V πi ( s ) and Q π i ( s , a ) here contain both intrinsic and extrinsic rewards . 3.1 EXPLORATION VIA INFORMATION-THEORETIC INFLUENCE . One critical problem in our learning framework presented above is to define the influence value I . For simplicity , we start with a two-agent case . The first method we propose is to use mutual information between agents ’ trajectories to measure one agent ’ s influence on other agents ’ learning processes . Such mutual information can be defined as information gain of one agent ’ s state transition given the other ’ s state and action . Without loss of generality , we define it from the perspective of agent 1 : MIπ2|1 ( S ′ 2 ; S1 , A1|S2 , A2 ) = ∑ s , a , s′2∈ ( S , A , S2 ) pπ ( s , a , s′2 ) [ log p π ( s′2|s , a ) − log pπ ( s′2|s2 , a2 ) ] , ( 5 ) where s = ( s1 , s2 ) is the joint state , a = ( a1 , a2 ) is the joint action , and Si and Ai are the random variables of state and action of agent i subject to the distribution induced by the joint policy π . So we define Iπ2|1 as MI π 2|1 ( S ′ 2 ; S1 , A1|S2 , A2 ) that captures transition interactions between agents . Optimizing this objective encourages agent 1 to visited critical points where it can influence the transition probability of agent 2 . We call such an exploration method exploration via informationtheoretic influence ( EITI ) . Optimizing MIπ2|1 with respect to the policy parameters θ1 of agent 1 is a little bit challenging , because it is an expectation with respect to a distribution that depends on θ1 . The gradient consists of two terms : ∇θ1MIπ ( S′2 ; S1 , A1|S2 , A2 ) = ∑ s , a , s′2∈ ( S , A , S2 ) ∇θ1 ( pπ ( s , a , s′2 ) ) log p ( s′2|s , a ) pπ ( s′2|s2 , a2 ) + ∑ s , a , s′2∈ ( S , A , S2 ) pπ ( s , a , s′2 ) ∇θ1 log p ( s′2|s , a ) pπ ( s′2|s2 , a2 ) . ( 6 ) While the second term is an expectation over the trajectory and can be shown to be zero ( see Appendix B.1 ) , it is unwieldy to deal with the first term because it requires the gradient of the stationary distribution , which depends on the policies and the dynamics of the environment . Fortunately , the gradient can still be estimated purely from sampled trajectories by drawing inspiration from the proof of the policy gradient theorem ( Sutton et al. , 2000 ) . The resulting policy gradient update is : ∇θ1Jθ1 ( t ) = ( R̂t1 − V̂ π1 ( st ) ) ∇θ1 log πθ1 ( at1|st1 ) ( 7 ) where V̂ π1 ( st ) is an augmented value function of R̂ t 1 = ∑h t′=t r̂ t′ 1 and r̂t1 = r t + ut1 + β log p ( st+12 |st1 , st2 , at1 , at2 ) p ( st+12 |st2 , at2 ) . ( 8 ) The third term , which we call EITI reward , is 0 when the agents are transition-independent , i.e. , when p ( st+12 |st1 , st2 , at1 , at2 ) = p ( s t+1 2 |st2 , at2 ) , and is positive when st1 , at1 increase the probability of agent 2 translating to st+12 . Therefore , the EITI reward is an intrinsic motivation that encourages agent 1 to visit more frequently the state-action pairs where it can influence the trajectory of agent 2 . The estimation of p ( st+12 |st1 , st2 , at1 , at2 ) and p ( s t+1 2 |st2 , at2 ) are discussed in Appendix C. We assume that agents know the states and actions of other agents , but this information is only available during centralized training . When execution , agents only have access to their local observations .

Summary: This paper studies the problem of designing effective exploration strategies in multi-agent domains. The key idea is to define one agent's exploration in terms of its interactions with other agents. This leads to two auxiliary exploration objectives, which measure how one agent's actions affect the dynamics and value of another agent's actions. The paper does an admirable job comparing the proposed method against a number of baselines, where the proposed method performs significantly better. Visualizations and ablation experiments nicely illustrate the contributions of various components of the method.