Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has been applied to solve various challenging problems , where an agent typically learns to maximize the expected sum of extrinsic rewards gathered as a result of its actions performed in the environment ( Sutton et al. , 1998 ) . Multi-Agent Reinforcement Learning ( MARL ) refers to the task of training a set of agents to maximize collective and/or individual rewards , while existing in the same environment and interacting with each other . Recent works have shown that agents with coordinated behaviors learn remarkably faster ( Roy et al. , 2019 ) since coordination helps the discovery of effective policies in cooperative tasks . Nevertheless , achieving coordination among agents still remains a central challenge in MARL ( Jaques et al. , 2019 ) . Prominent works often resort to a popular learning paradigm called Centralized Training with Decentralized Execution ( CTDE ) ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , where each agent is evaluated using a centralized critic and has access to extra information about the policies of other learning agents during training . At the time of execution , policies ’ actions are restricted to local information only ( i.e . their own observations ) . Our work is primarily motivated by the following natural questions : In multi-agent settings , how can we quantify the effect of an agent ( or player ) on other teammates ’ behaviors ( particularly on their performance ) ? And to what degree exploiting this quantity can lead to coordinated behaviors and consequently better overall performances ? To that end , we propose a novel approach that aims at promoting coordination for cooperative tasks by augmenting CTDE MARL main return-maximization objective with an additional multi-agent objective that acts as a policy regularizer ; we refer to the latter objective as the influence function . To build intuition , a chosen agent , which we call the “ influencer ” , assesses the progress that other agents are making given its current policy and consequently learns behaviors that will result in an improved performance of its teammates . Concretely , we formulate the influence of an influencer π as an estimation of the dissimilarity between other agents ’ behaviors and their targets given the current behavior of π . The influencer is encouraged to learn behaviors that are expected to minimize that dissimilarity . We also propose two approaches to estimate the influence and empirically show that they yield unbiased estimates of the true value . Agents acting upon the proposed coordination paradigm learn to efficiently exploit the observed joint action space using available information . However , and since the joint space grows exponentially with the number of agents , it is highly unlikely that agents will have access to sufficient information to learn optimal behaviors to solve the task at hand ; this problem arises in many scenarios such as sparse-reward environments , thus a proper exploration scheme is often required . However , many existing multi-agent deep reinforcement learning algorithms still use mostly noise-based techniques ( Liu et al. , 2021 ; Rashid et al. , 2018 ; Yang et al. , 2018 ) . Moreover , independent exploration proved to be inefficient in cooperative settings ( Roy et al. , 2019 ) . Recently , this challenge was addressed through Intrinsic Motivation ( IM ) ( Jaques et al. , 2019 ; Du et al. , 2019 ; Zhou et al. , 2020b ) . Many approaches employ IM to encourage exploration of state-space ( Han et al. , 2020 ; Burda et al. , 2019 ) or state-action space ( Fayad & Ibrahim , 2021 ) by identifying novel configurations and rewarding an agent for visiting them . We provide an extension of these ideas into multi-agent settings and further build connection between reward shaping and coordinated behavior learning , where we choose an agent to act as an influencer ( i.e . regularize its standard objective using the influence function ) while other agents learn to maximize the expected sum of both extrinsic and intrinsic rewards . To sum up , our main contributions are threefold : 1 ) developing an influence function to promote learning coordinated behaviors and improve team performance ; 2 ) extending exploration via random network distillation to multi-agent settings by crafting a `` novelty '' function that rewards underexplored behaviors ; 3 ) formulating a novel intrinsic incentive to promote learning diverse team behaviors to help uncover complex behaviors in a collaborative way . We demonstrate the effectiveness of our methods on a comprehensive set of challenging tasks which include , but not limited to , the StarCraft Multi-Agent Challenges ( SMAC ) ( Samvelyan et al. , 2019 ) and the Multi-Agent Particle Environments ( MAPE ) ( Mordatch & Abbeel , 2018 ; Lowe et al. , 2017 ) . Empirical results show a significant improvement over a wide variety of state-of-the-art MARL approaches . We also conduct insightful ablation studies to understand the relative importance of each component of the approach individually . 2 BACKGROUND . 2.1 MARKOV GAMES . Markov games ( Littman , 1994 ; Agarwal et al. , 2020 ; Daskalakis et al. , 2021 ) are a superset of Markov decision process ( MDPs ) and matrix games , including both multiple agents and multiple states . Formally , we consider a setting with N > 2 agents ( [ N ] : = { 1 , 2 . . . , N } ) who repeatedly select actions in a shared MDP . The goal of each agent is to maximize their respective action value function . Formally , an MDP is defined as a tuple G = ( S , N , { Ai , ri } i∈ [ N ] , T , γ , ρ ) , where : • S is the finite states space , where the initial states are determined by a distribution ρ : S → [ 0 , 1 ] . Since we consider partially observable testing environments , we denote by si the information observed by agent i about the global state s ∈ S. • Ai is a finite action space for agent i with generic element ai ∈ Ai . Using common conventions , we will write A = Πj∈ [ N ] Aj and A−i = Πj 6=iAj to denote the joint action spaces of all agents and of all agents other than i with generic elements a = ( ai ) i∈ [ N ] and a−i = ( aj ) i 6=j∈ [ N ] , respectively . According to this notation , we have that a = ( ai , a−i ) ∀i . • ri : S × A → R is the individual reward function of agent i , i.e. , ri ( s , ai , a−i ) is the instantaneous reward of agent i when agent i takes action ai and all other agents take actions a−i at state s ∈ S. • T is the transition probability function , for which T ( s′|s , a ) is the probability of transitioning from s to s′ when a ∈ A is the action profile chosen by the agents . • γ is a discount factor for future rewards of the MDP , shared by all agents . In a Markov game , each agent is independently choosing actions and receiving rewards . Conventionally , an agent k aims to maximize its own total expected return Rk = ∑T t=0 γ trk , t where T is the time horizon . 2.2 MULTI-AGENT DEEP DETERMINISTIC POLICY GRADIENT . MADDPG ( Lowe et al. , 2017 ) is a multi-agent extension of the DDPG algorithm ( Lillicrap et al. , 2015 ) . It adapts the CTDE paradigm , where each agent i possesses its own deterministic policy µ ( i ) for action selection and criticQ ( i ) for state-action value estimation , respectively parameterized by θ ( i ) and φ ( i ) . All parametric models are trained off-policy from previous transitions ζt : = ( st , at , rt , st+1 ) uniformly sampled from a replay buffer D. Each centralized critic is trained to estimate the expected return for a particular agent i from the Q-learning loss : L ( i ) ( φ ( i ) ) = Eζt∼D [ ∥∥∥Q ( i ) ( st , at ; φ ( i ) ) − [ ri , t + γQ ( i ) target ( st+1 , µ ( st+1 ) ) ] ∥∥∥2 ] ( 1 ) Each policy is updated to maximize the expected discounted return of the corresponding agent i : J ( i ) PG ( θ ( i ) ) = ED [ Q ( i ) ( st , at ) ] ( 2 ) Notice that while optimizing an agent ’ s policy , all agents ’ observation-action pairs are taken into consideration . By that , the value functions of all agents are trained in a centralized , stationary environment , despite happening in a multi-agent setting . Moreover , this procedure allows for the learning of coordinated strategies , yet needs to be augmented with efficient exploration methods that reward novel action configurations which may lead to the discovery of higher-return behaviors . 3 METHODS . 3.1 BASIC INFLUENCE . Intuitively , one can define coordination in a team of agents as the behavior of each individual agent being informed by other agents . Furthermore , agents ’ behaviors can be inter-affected either directly through communication for example or indirectly through task-specific shared goals and/or rewards or the dynamics of the environment . We hypothesize that when agents learn in a cooperative setting , they tend to affect each other ’ s exploitation processes , we confirm the hypothesis throughout the paper and build on that to formalize a general method to foster influential interactions and learn meaningful coordination protocols . Specifically , consider a setting where N agents with policies π1 , π2 , . . . , πN . We assume that the transition dynamics are dependent on agents ’ actions ( i.e . T ( s′|s , a ) 6= T ( s|s′ ) ) , and use tasks that satisfy this assumption to validate our methods . We will write π , π−j to denote the joint policy of all agents and of all agents other than agent j , respectively . For two fixed agents i and j , the influence of the behavior of j ’ s policy on i ’ s performance ( expected return ) can be quantified by `` marginalizing '' out all effects induced by other agents on the expected outcomes of i . More precisely , if the desired quantity is qπji , then for all ( s , aj ) ∈ S ×Aj : q πj i ( s , aj ) = E a−j∼π−j [ ri ( s , a ) ] + γ ∑ s′ p ( s′|s , aj ) · E a′j∼πj ( .|s ′ j ) [ q πj i ( s ′ , a′j ) ] p ( s′|s , aj ) = ∑ a−j T ( s′|s , a ) ∏ k 6=j πk ( ak|s ) ( 3 ) One can note that , analogously to the action-value function Qi , q πj i measures tuple values w.r.t i where the learning dynamics of i conditioned only on the behavior of j . Naturally , in expectation , q πj i represents the sum of rewards that an agent i is expected to receive when influenced by πj . Based on above , we say that an agent is an optimal influencer when agents only influenced by its policy learn optimal behaviors . To that end , if we fix an agent ( say 1 ) , then its influence , F ( π1 ) , on the rest of the team is defined as follows : F ( π1 ) = ∑ 2≤i≤N αi · Es , a1∼π1 [ ( qπ1i ( s , a1 ) −maxu Qi ( s , u ) ) 2 ] ( 4 ) where ( αi ) 2≤i≤N are positive scaling factors that sum up to 1 and Qi is the action value function of agent i . Intuitively , since F ( π1 ) is the difference between the expected rewards of other agents conditioned on the current behavior of π1 and the maximum returns that can be achieved by other agents in the given task , minimizing F will encourage π1 to take actions that lead other agents ’ conditioned returns ( qπ1i ) 2≤i≤N closer to their maximum unrestricted returns ( maxuQi ( s , u ) ) , which matches our definition of an optimal influencer . Hence , the objective functions of agents i ∈ [ N ] are : Ji ( θi ) = Eπ [ ∑ 0≤t≤T γ tri , t ] − λF ( π1 ) 1i=1 , where θi are the parameters of πi and λ is a hyperparameter called the influence importance temperature . Theorem 1 . ( Influence Gradient ) ∇θ1F ( π1 ) = ∑ 2≤i≤N αiEs , π1 [ ∇θ1 log π1 ( a1|s ) g ( s , a1 ) 2 + 2g ( s , a1 ) Es , π1 [ ∇θ1 log π1 ( a1|s ) q π1 i ( s , a1 ) ] ] ( 5 ) where g ( T ) = g ( s , a1 ) = qπ1i ( s , a1 ) −maxuQi ( s , u ) . It has been shown empirically that the existence of one influencer can greatly improve the overall team performance . Thus , we only consider one influencer throughout our practical sections and upon that , we call our method `` Asymmetric Learning for Influencing a Team of Agents ( ALITA ) '' .

Summary: This paper introduces two techniques for Centralized Training with Decentralized Execution (CTDE) MARL. Specifically: (1) It proposes an "influence" training objective by which one agent is encouraged to help the other agents reach target returns. (2) It introduces intrinsic motivation and intrinsic cost terms aimed at encouraging efficient joint exploration of the Markov game. It then goes on to give several empirical comparisons with baselines, showing the utility of the approach in Starcraft Multi-Agent Challenge (SMAC), two sparse reward settings, Multi-Agent Particle Environments, and OpenAI Gym continuous control environments. It also performs an ablation study in (SMAC), demonstrating the usefulness of each component of the proposed method.