Intrinsic Motivation for Encouraging Synergistic Behavior

1 INTRODUCTION . Consider a multi-agent environment such as a team of robots working together to play soccer . It is critical for a joint policy within such an environment to produce synergistic behavior , allowing multiple agents to work together to achieve a goal which they could not achieve individually . How should agents learn such synergistic behavior efficiently ? A naive strategy would be to learn policies jointly and hope that synergistic behavior emerges . However , learning policies from sparse , binary rewards is very challenging – exploration is a huge bottleneck when positive reinforcement is infrequent and rare . In sparse-reward multi-agent environments where synergistic behavior is critical , exploration is an even bigger issue due to the much larger action space . A common approach for handling the exploration bottleneck in reinforcement learning is to shape the reward using intrinsic motivation , as was first proposed by Schmidhuber ( 1991 ) . This has been shown to yield improved performance across a variety of domains , such as robotic control tasks ( Oudeyer et al. , 2007 ) and Atari games ( Bellemare et al. , 2016 ; Pathak et al. , 2017 ) . Typically , intrinsic motivation is formulated as the agent ’ s prediction error regarding some aspects of the world ; shaping the reward with such an error term incentivizes the agent to take actions that “ surprise it , ” and is intuitively a useful heuristic for exploration . But is this a good strategy for encouraging synergistic behavior in multi-agent settings ? Although synergistic behavior may be difficult to predict , it could be equally difficult to predict the effects of certain single-agent behaviors ; this formulation of intrinsic motivation as “ surprise ” does not specifically favor the emergence of synergy . In this paper , we study an alternative strategy for employing intrinsic motivation to encourage synergistic behavior in multi-agent tasks . Our method is based on the simple insight that synergistic behavior leads to effects which would not be achieved if the individual agents were acting alone . So , ∗Work done during an internship at Facebook AI Research . we propose to reward agents for joint actions that lead to different results compared to if those same actions were done by the agents individually , in a sequential composition . For instance , consider the task of twisting open a water bottle , which requires two hands ( agents ) : one to hold the base in place , and another to twist the cap . Only holding the base in place would not effect any change in the bottle ’ s pose , while twisting the cap without holding the bottle in place would cause the entire bottle to twist , rather than just the cap . Here , holding with one hand and subsequently twisting with the other would not open the bottle , but holding and twisting concurrently would . Based on this intuition , we propose a formulation for intrinsic motivation that leverages the difference between the true effect of an action and the composition of individual-agent predicted effects . We then present a second formulation that instead uses the discrepancy of predictions between a joint and a compositional prediction model . While the latter formulation requires training a forward model alongside learning the control strategy , it has the benefit of being analytically differentiable with respect to the action taken . We later show that this can be leveraged within the policy gradient framework , in order to obtain improved sample complexity over using the policy gradient as-is . As our experimental point of focus , we study six simulated robotic tasks : four bimanual manipulation ( bottle opening , ball pickup , corkscrew rotating , and bar pickup ) and two multi-agent locomotion ( ant push and soccer ) . All tasks have sparse rewards : 1 if the goal is achieved and 0 otherwise . These tasks were chosen both because they require synergistic behavior , and because they represent challenging control problems for modern state-of-the-art deep reinforcement learning algorithms ( Levine et al. , 2016 ; Lillicrap et al. , 2016 ; Gu et al. , 2017 ; Mnih et al. , 2016 ; Nagabandi et al. , 2018 ) . Across all tasks , we find that shaping the reward via our formulation of intrinsic motivation yields more efficient learning than both 1 ) training with only the sparse reward signal and 2 ) shaping the reward via the more standard single-agent formulation of intrinsic motivation as “ surprise , ” which does not explicitly encourage synergistic behavior . We view this work as a step toward general-purpose synergistic multi-agent reinforcement learning . 2 RELATED WORK . Prediction error as intrinsic motivation . The idea of motivating an agent to reach areas of the state space which yield high model prediction error was first proposed by Schmidhuber ( 1991 ) . Generally , this reward obeys the form ‖f ( x ) − f̂ ( x ) ‖ , i.e . the difference between the predicted and actual value of some function computed on the current state , the taken action , etc . ( Barto , 2013 ; Oudeyer et al. , 2007 ; Bellemare et al. , 2016 ) ; intrinsic motivation can even be used on its own when no extrinsic reward is provided ( Pathak et al. , 2017 ; 2019 ; Burda et al. , 2019 ; Haber et al. , 2018 ) . A separate line of work studies how agents can synthesize a library of skills via intrinsic motivation in the absence of extrinsic rewards ( Eysenbach et al. , 2019 ) . Recent work has also studied the use of surprise-based reward to solve gentle manipulation tasks , with the novel idea of rewarding the agent for errors in its own predictions of the reward function ( Huang et al. , 2019 ) . In this paper , we will propose formulations of intrinsic motivation that are geared toward multi-agent synergistic tasks . Exploration in multi-agent reinforcement learning . The problem of efficient exploration in multi-agent settings has received significant attention over the years . Lookahead-based exploration ( Carmel & Markovitch , 1999 ) is a classic strategy ; it rewards an agent for exploration that reduces its uncertainty about the models of other agents in the environment . More recently , social motivation has been proposed as a general principle for guiding exploration ( Jaques et al. , 2019 ) : agents should prefer actions that most strongly influence the policies of other agents . LOLA ( Foerster et al. , 2018 ) , though not quite an exploration strategy , follows a similar paradigm : an agent should reason about the impact of its actions on how other agents learn . Our work approaches the problem from a different angle that incentivizes synergy : we reward agents for taking actions to affect the world in ways that would not be achieved if the agents were acting alone . Bimanual manipulation . The field of bimanual , or dual-arm , robotic manipulation has a rich history ( Smith et al. , 2012 ) as an interesting problem across several areas , including hardware design , model-based control , and reinforcement learning . Model-based control strategies for this task often draw on hybrid force-position control theory ( Raibert et al. , 1981 ) , and rely on analytical models of the environment dynamics , usually along with assumptions on how the dynamics can be approximately decomposed into terms corresponding to the two arms ( Hsu , 1993 ; Xi et al. , 1996 ) . On the other hand , learning-based strategies for this task often leverage human demonstrations to circumvent the challenge of exploration ( Zollner et al. , 2004 ; Gribovskaya & Billard , 2008 ; Kroemer et al. , 2015 ) . In this work , we describe an exploration strategy based on intrinsic motivation . 3 APPROACH . Our goal is to enable learning for synergistic tasks in settings with sparse extrinsic rewards . A central hurdle in such scenarios is the exploration bottleneck : there is a large space of possible action sequences that the agents must explore in order to see rewards . In the absence of intermediate extrinsic rewards to guide this exploration , one can instead rely on intrinsic rewards that bias the exploratory behavior toward “ interesting ” actions , a notion which we will formalize . To accomplish any synergistic task , the agents must work together to affect the environment in ways that would not occur if they were working individually . In Section 3.1 , we present a formulation for intrinsic motivation that operationalizes this insight and allows guiding the exploration toward synergistic behavior , consequently learning the desired tasks more efficiently . In Section 3.2 , we present a second formulation that is ( partially ) differentiable , making learning even more efficient by allowing us to compute analytical gradients with respect to the action taken . Finally , in Section 3.3 we show how our formulations can be used to efficiently learn task policies . Problem Setup . Each of the tasks we consider can be formulated as a two-agent finite-horizon MDP ( Puterman , 1994 ) .1 We denote the environment as E , and the agents as A and B . We assume a state s ∈ S can be partitioned as s : = 〈sA , sB , senv〉 , where sA ∈ SA , sB ∈ SB , and senv ∈ Senv . Here , sA and sB denote the proprioceptive states of the agents , such as joint configurations of robot arms , and senv captures the remaining aspects of the environment , such as object poses . An action a ∈ A is a tuple a : = 〈aA , aB〉 , where aA ∈ AA and aB ∈ AB , consisting of each agent ’ s actions . We focus on settings where the reward function of this MDP is binary and sparse , yielding reward rextrinsic ( s ) = 1 only when s achieves some desired goal configuration . Learning in such a setup corresponds to acquiring a ( parameterized ) policy πθ that maximizes the expected proportion of times that a goal configuration is achieved by following πθ . Unfortunately , exploration guided only by a sparse reward is challenging ; we propose to additionally bias it via an intrinsic reward function . Let s̄ ∼ E ( s , a ) be a next state resulting from executing action a in state s. We wish to formulate an intrinsic reward function rintrinsic ( s , a , s̄ ) that encourages synergistic actions and can thereby enable more efficient learning . 3.1 COMPOSITIONAL PREDICTION ERROR AS AN INTRINSIC REWARD . We want to encourage actions that affect the environment in ways that would not occur if the agents were acting individually . To formalize this notion , we note that a “ synergistic ” action is one where 1Our problem setup and proposed approach can be extended to settings with more than two agents . Details , with accompanying experimental results , are provided in Section 4.5. the agents acting together is crucial to the outcome ; so , we should expect a different outcome if the corresponding actions were executed sequentially , with each individual agent acting at a time . Our key insight is that we can leverage this difference between the true outcome of an action and the expected outcome with individual agents acting sequentially as a reward signal . We can capture the latter via a composition of forward prediction models for the effects of actions by individual agents acting separately . Concretely , let fA : Senv × SA × AA → Senv ( resp . fB ) be a singleagent prediction model that regresses to the next environment state resulting from A ( resp . B ) taking an action in isolation.2 We define our first formulation of intrinsic reward , rintrinsic1 ( s , a , s̄ ) , by measuring the prediction error of s̄env using a composition of these single-agent prediction models : f composed ( s , a ) = fB ( fA ( senv , sA , aA ) , sB , aB ) , rintrinsic1 ( s , a , s̄ ) = ‖s̄env − f composed ( s , a ) ‖ . For synergistic actions a , the prediction f composed ( s , a ) will likely be quite different from s̄env . In practice , we pretrain fA and fB using data of random interactions in instantiations of the environment E with only a single active agent . This implies that the agents have already developed an understanding of the effects of acting alone before being placed in multi-agent environments that require synergistic behavior . Note that while random interactions sufficed to learn useful prediction models fA and fB in our experiments , this is not essential to the formulation , and one could leverage alternative single-agent exploration strategies to collect interaction samples instead .

Summary: The paper proposes a novel algorithm for encouraging synergistic behavior in multi-agent setups with an intrinsic reward that promotes the agents to work together to achieve states that they cannot achieve individually without cooperation. The paper focuses on a two-agent environment where an approximate forward dynamics model is learnt for each agent, and can be composed sequentially to predict the next environment state given each agent’s action. However, this prediction will be inaccurate if the agent’s affected the environment state in such a way that individual dynamics model cannot predict i.e. synergistic behavior was produced. This prediction error is used as extrinsic reward by the proposed approach, while also having a variant where the true next state is replaced by another approximation of a joint forward model which allows for differentiability of actions with respect to the intrinsic reward. Empirical analysis shows that this intrinsic reward promotes synergetic behavior on two-agent robotic manipulation tasks and achieves better performance that baselines and ablations.