Intrinsic Motivation for Encouraging Synergistic Behavior
Authors: Rohan Chitnis
Shubham Tulsiani
Saurabh Gupta
Abhinav Gupta
Venue:  (2020)
Keywords: reinforcement learning
intrinsic motivation
synergistic
robot manipulation

Abstract: We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own. Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent. We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy. While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken. We validate our approach in robotic bimanual manipulation and multi-agent locomotion tasks with sparse rewards; we find that our approach yields more efficient learning than both 1) training with only the sparse reward and 2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior. Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.

Published as a conference paper at ICLR 2020
INTRINSIC MOTIVATION FOR ENCOURAGING SYNERGISTIC BEHAVIOR
Rohan Chitnis
Shubham Tulsiani
Saurabh Gupta
Abhinav Gupta
MIT Computer Science and Artificial Intelligence Laboratory, Facebook Artificial Intelligence Research ronuchi souchit@man comit.edu,, shubhtulss fb. com, saurabhga 11 11liol eddu, gabhinav: com
ABSTRACT
We study the role of intrinsic motivation exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve goal they could not individually. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own. Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via composition of the pre- dicted effect for each individual agent. We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy. While the former is simpler, the latter has the henefit of being analytically differentiable with respect the action taken. We validate our approach in robotic bimanual manipulation and multi-agent lo- comotion tasks with sparse rewards; we find that our approach yields more ef- ficient learning than both 1) training with only the sparse reward and 2) using the typical surprise-base based formulation of intrinsic motivation, which does not bias toward synergistic behavior. Videos available the project webpage: http://///e//////1110000////////
prctps://stte
INTRODUCTION
Consider multi-agent environment such as team of robots working together to play soccer. It critical for a joint policy within such an environment to produce synergistic behavior, allowing multiple agents work together achieve a goal which they could achieve individually. How should agents learn such synergistic behavior efficiently? A naive strategy would be to learn policies jointly and hope that synergistic behavior emerges. However, learning policies from sparse, binary rewards is very challenging exploration is a huge bottleneck when positive reinforcement is in- frequent and rare. sparsc-reward multi-agent environments where synergistic behavior is critical, exploration is an even bigger issue due to the much larger action space.
common approach for handling the exploration bottleneck in reinforcement learning to shape the reward using intrinsic motivation, as was first proposed by [Schmidhuber (1991). This has been shown to yield improved performance across a variety of domains, such as robotic control tasks (Oudeyer et al. (2077) and Atari games (Bellemare et al.) [016)] Pathak et al. (017). Typically, intrinsic motivation is formulated as the agent's 's prediction error regarding some aspects the world; shaping the reward with such an error term incentivizes the agent to take actions that "surprise it," and is intuitively a useful heuristic for exploration. But this a good strategy for encouraging syn- ergistic behavior in multi-agent settings? Although synergistic behavior may be difficult predict, could be equally difficult to predict the effects of certain single-agent behaviors; this formulation of intrinsic motivation as "surprise" does not specifically favor the emergence of synergy.
In this paper, we study an alternative strategy for employing intrinsic motivation to encourage syn- ergistic behavior in multi-agent tasks. Our method is based the simple insight that synergistic behavior leads to effects which would not be achieved if the individual agents were acting alone. So,
Work done during intersship Facebook AT Research.

---

Published as a conference paper at ICLR 2020
floint
predicted next state (e.g. bar fully lifted)
state (e-g. bar on table)
intrinsic reward for training iig
ycompused
distance
(pretrained)
predicted next state (e.g. bar partiolly lifted)
loss training plaint
action = ( a
distance
Environment
true next state
Figure I: An overview of our approach to incentivizing synergistic behavior via intrinsic motivation. A heavy red bar (requiring bar from opposite ends. A composition
arms to lift) rests table, and the policy wy suggests for arms A and B lift the
pretrained single-agent forward models, f and f predicts the resulting state to be one where the bar is only partially lifted, since neither f nor f has ever encountered states where the bar is lifted during training. A forward model trained on the complete two-agent environment, foour correctly predicts that the bar is fully lifted, very different from the compositional prediction. We train we to prefer actions such as these, as way bias toward synergistic behavior. Note that differentiating this intrinsic reward with respect to the action taken does not require differentiating through the environment.
we propose to reward agents for joint actions that lead to different results compared to if those same actions were done by the agents individually, in sequential composition. For instance, consider the task of twisting open a water bottle, which requires two hands (agents): one to hold the base in place, and another to twist the cap. Only holding the base in place would not effect any change in the bottle's pose, while twisting the cap without holding the bottle in place would cause the entire bottle to twist, rather than just the cap. Here, holding with one hand and subsequently twisting with the other would not open the bottle. but holding and twisting concurrently would.
Based on this intuition, we propose a formulation for intrinsic motivation that leverages the differ- ence between the true effect of an action and the composition of individual-agen predicted effects. We then present second formulation that instead uses the discrepancy of predictions between a joint and a compositional prediction model. While the latter formulation requires training a forward model alongside learning the control strategy, it has the benefit of being analytically differentiable with respect to the action taken. We later show that this can be leveraged within the policy gradient improved improved sample complexity over using the policy gradient as-is.
framework, in order to obtain
As experimental point of focus, we study six simulated robotic tasks: four bimanual manip- ulation (bottle opening, ball pickup, corkscrew rotating, and bar pickup) and two multi-agent lo- comotion (ant push and soccer). All tasks have sparse rewards: 1 if the goal is achieved and 0 otherwise. These tasks were chosen both because they require synergistic behavior, and because they represent challenging control problems for modern ofate-offree the-artttttr art deep reinforcement learning algorithms (Levine et al., 2016; Lillicrap et al. [2016] Gu et al. 2017 Mnih et al. 2016 Nagabandi al. 2018). Across all tasks, find that shaping the reward via our formulation of intrinsic moti- vation yields more efficient learning than both 1) training with only the sparse reward signal and 2) shaping the reward via the more standard single-agent formulation of intrinsic motivation as "sur- prise," which does not explicitly encourage synergistic behavior. We view this work as step toward general- purposse synergistic multi-agent reinforcement learning.
RELATED WORK
Prediction error intrinsic motivation. The idea of motivating an agent reach areas of the state space which yield high model prediction error was first proposed by Schmidhuber (1991), Generally, this reward obeys the form [z) - fxx)]. i.c. the difference between the predicted and actual value of some function computed on the current state, the taken action, etc. (Barto 2013 Oudeyer et al. 2007 Bellemare et al.] 2016): intrinsic motivation can even be used on its own when no extrinsic reward is provided (Pathak et al., 2017 2019 Burda et al. 2019 Haber et al. 2018). A separate line of work studies how agents can synthesize library of skills via intrinsic motivation in the absence of extrinsic rewards (Eysenbach et al. 2019). Recent work has also studied the use of surprise- based reward to solve gentle manipulation tasks, with the novel idea of rewarding the agent for errors in its own predictions of the reward function (Huang et al. 2019). In this paper, we will propose formulations of intrinsic motivation that are geared toward multi-agent synergistic tasks.
2

---

Published as a conference paper at ICLR 2020
Exploration in multi-agent reinforcement learning. The problem of efficient exploration in multi-agent settings has received significant attention over the years. Lookahead based explo- ration (Carmel & Markovitch, 1999] is classic strategy; it rewards agent for exploration that reduces its uncertainty about the models of other agents in the environment. More recently, social motivation has been proposed as a general principle for guiding exploration (Jaques et al.) 2019): agents should prefer actions that most strongly influence the policies of other agents. LOLA (Foerster ct al. 2018), though not quite an exploration strategy, follows similar paradigm: an agent should reason about the impact of its actions on how other agents learn. Our work approaches the problem from different angle that incentivizes synergy: we reward agents for taking actions to affect the world in ways that would not be achieved if the agents were acting alone.
Bimanual manipulation. The field of bimanual, or dual-arm, robotic manipulation has a rich his- tory (Smith et al., (012)) as an interesting problem across several areas, including hardware design, model-based control, and reinforcement learning. Model-based control strategies for this task often draw on hybrid force-position control theory (Raibert et al. 198] I), and rely on analytical models of the environment dynamics, usually along with assumptions on how the dynamics can be approxi- mately decomposed into terms corresponding to the two arms (Hsu 1993, Xi et al. 1996). On the other hand, learning- barning-based strategies for this task often leverage human demonstrations to circum- vent the challenge of exploration (Zoliner al. [004 Gribovskaya & Billard 2008; Kroemer al., 2013). In this work, we describe an exploration strategy based on intrinsic motivation.
3 APPROACH
Our goal is to enable learning for synergistic tasks in settings with sparse extrinsic rewards. A central hurdle in such scenarios is the exploration bottleneck: there is a large space of possible action sequences that the agents must explore in order to see rewards. In the absence of intermediate extrinsic rewards to guide this exploration, one instead rely on intrinsic rewards that bias the exploratory behavior toward "interesting" actions, a notion which we will formalize.
To accomplish any synergistic task, the agents must work together to affect the environment in ways that would not occur if they were working individually. In Section 3.1, we present formulation for intrinsic motivation that operationalizes this insight and allows guiding the exploration toward synergistic behavior, consequently learning the desired tasks more efficiently. In Section 3.2 we present a second formulation that (partially) differentiable, making learning even more efficient by allowing us to compute analytical gradients with respect to the action taken. Finally, in Section 3.3 we show how our formulations can be used to efficiently learn task policies.
Problem Setup. Each of the tasks we consider can be formulated as a two-agent finite-horizon MDP (Puterman 1994 We denote the environment as E, and the agents as A and B. We assume a state 8 E S can be partitioned as == (s*, s sSmN), where & E S s E SB, and gon E Seav Here, 5 and s denote the proprioceptive states the agents, such as joint configurations of robot arms, and e captures the remaining aspects of the environment, such as object poses. An action a E A is a tuple a == (a a a)), where a E A and B E AA consisting of each agent's actions.
We focus settings where the reward function of this MDP is binary and sparse, yielding reward extrinsic (s) = 1 only when achieves some desired goal configuration. Learning in such a setup corresponds acquiring a (parameterized) policy me that maximizes the expected proportion of times that a goal configuration is achieved by following T
Unfortunately, exploration guided only by sparse reward is challenging; we propose to additionally bias it via an intrinsic reward function. Let & ~ E(s, a) be a next state resulting from executing action a in state s. We wish formulate an intrinsic reward function pintrinsic (s, a, 3) that encourages synergistic actions and can thereby enable more efficient learning.
3.1 COMPOSITIONAL PREDICTION ERROR AS AN INTRINSIC REWARD
We want to encourage actions that affect the environment in ways that would not occur if the agents were acting individually. To formalize this notion, we note that a "synergistic" action is one where
!Our problem setup and proposed approach can he extended to settings with more than two agents. Details, with accompanying experimental results, are provided in Section 4.5
3

---

Published as a conference paper at ICLR 2020
the agents acting together is crucial to the outcome; so, should expect different outcome if the corresponding actions were executed sequentially, with each individual agent acting at a time.
Our key insight is that can leverage this difference between the true outcome of action and the expected outcome with individual agents acting sequentially as a reward signal. We can capture the latter via a composition of forward prediction models for the effects of actions by individual agents acting separately. Concretely, let f San x S x A + Senv (resp. f8) be a single- the next environment state resulting from A (resp. B) taking an action in isolation 2 We define our first formulation of intrinsic reward, r (s, a, 8), by measuring the prediction error of senh using a composition of these single-agent prediction models:
agent prediction model that regresses
prompusd (s,a) (f SA , ". ,²,a")) aa
plntrinsic (s, a, 5) || frooppseedd (s, a)ll-
For synergistic actions the prediction foomposed (s, a) will likely be quite different from senv.
In practice, we pretrain í and l using data of random interactions in instantiations of the envi- ronment & with only single active agent. This implies that the agents have already developed an understanding of the effects of acting alone before being placed in multi-agent environments that require synergistic behavior. Note that while random interactions sufficed to Iearn useful predic- tion models f and f in our experiments, this is not essential to the formulation, and one could leverage alternative single-agent exploration strategies to collect interaction samples instead.
3.2 PREDICTION DISPARITY AS A DIFFERENTIABLE INTRINSIC REWARD
The reward ri ever, note that this "measurement of synergy" for action in state requires explicitly observing the outcome & of executing a in the environment. In contrast, when humans reason about synergistic tasks such twisting open bottle cap while holding the bottle base, we judge whether actions will have a synergistic effect without needing to execute them to make this judgement. Not only is the non-dependence of the intrinsic reward on scientifically interesting, but it is also practically desir- able. Specifically, the term /composed (s. (s.a) is analytically differentiable with respect to (assuming that one uses differentiable regressors f and f such as neural networks), but genv is not, since 3 depends on a via the black-box environment. If we can reformulate the intrinsic reward to be analytically differentiable with respect to a, we can leverage this for more sample-efficient learning.
riitrinsic (s, a, 3) presented above encourages actions that have a synergistic effect. How-
To this end, we observe that our formulation rewards actions where the expected outcome under the compositional prediction differs from the outcome when the agents act together. While we used the observed state as the indication of "outcome when the agents act together," we could instead use predicted outcome here. We therefore additionally train a joint prediction model froin x A - SanN that, given the states and actions of both agents, and the environment state, predicts the next environment state. We then define our second formulation of intrinsic reward, r2 intmiiic (s, a, +), using the disparity between the predictions of the joint and compositional models:
intrinsic (s, o, = || pioint (8, a) - ccompssed (8, a)ll-
2
Note that there is no dependence on 8. At first, this formulation may seem less efficient than r doorinsii since floint can at best only match geny and requires being trained on data. However, we note that this formulation makes the intrinsic reward analytically diffcrentiable with respect to the action a executed; we can leverage this within the learning algorithm to obtain more informative gradient updates, as we discuss further in the next section.
Relation to Curiosity. Typical approaches to intrinsic motivation (Stadie et al. 2015 Pathak al. 2017) which reward an agent for "doing what surprises it," take on the form r noon-syneriitic ($, a, 3) [aom (s, a) ge These curiosity-based methods will encourage the system to keep finding new behavior that surprises it, and thus can be seen as a technique for curiosity-driven skill discovery. In contrast, we are focused on synergistic multi-agent tasks with an extrinsic (albeit sparse) reward, so our methods for intrinsic motivation are not intended to encourage a diversity of learned behaviors, but rather to bias exploration enable sample-efficient learning for a given task.
inrinsie
2 As the true environment dynamics are stochastic, it can be useful to consider probabilistic regressors f. However, recent successful applications of model-hased reinforcement learning (Nagabandi et al. [0188 Clav- era et al. 2019) have used deterministic regressors, modeling just the maximum likelihood transitions.
4

---

Published as a conference paper at ICLR 2020
3.3 LEARNING REAARR SYNERGISTIC TASKS
We simultaneously learn the joint prediction model fioint and the task policy ng. We train To via reinforcement learning to maximize the expected total shaped reward - = [1,2) + X pestrisic across an episode. Concurrently, we make dual-purpose use of the transition samples {(s, a, 3)} collected during the interactions with the environment to train fioms by minimizing the loss [ |- This simultaneous training of froint and Ta. as was also done by Stadie et al. 20015), obviates the need for collecting additional samples to pretrain flui and ensures that the joint prediction model is trained using the "interesting" synergistic actions being explored. Full pseudocode is provided in Appendix A
Our second intrinsic reward formulation allows us to leverage differentiability with respect to the action taken to make learning via policy gradient methods more efficient. Recall that any policy gra- dient algorithm (Schulman al., 2017; 2015; Williams 1992) performs gradient ascent with respect to policy parameters 0 on the expected reward over trajectories: J(0) = E, ( (r)]- Expanding,
we have J(f) = E, E ( ac. = E, EE 22 intrinsic (st, at, + ( where
T is the horizon. We show in Appendix B that the gradient can be written as:
VoJ(O) Σ B, [r (st, ac. log EG,
(
intrinsic (st, Res I]--
00, Here, := (8p. 0o, ..., St, 0z) denotes trajectory up to time i, and Fz == (so, ao. .... 81) denotes the
same but excluding a, Given state 8t, and assuming a differentiable way of sampling a, ~ (T:))). such using the reparameterization trick (Kingma & Welling 2014), we can analytically compute the inner gradient in the second term since 2 muumss (3,, ae, :) is differentiable with respect to ag (again, assuming the regressors fA, f and fivint are differentiable). In Equation [ the first term is similar to what typical policy gradient algorithms compute, with the difference being the use of pa (F(F)) instead of po (Tx); the intuition is that we should not consider the effects of ae here since it gets accounted for by the second term. In practice, however, we opt to treat the policy gradient algorithm as black box, and simply add (estimates of) the gradients given by the second term to the gradients yielded by the black-box algorithm. While this leads to double-counting certain gradients (those of the expected reward at each timestep with respect the action at that timestep), our preliminary experiments found this to minimally affect training, and make the implementation more convenient as one can leverage an off- off-the-shelf optimizer like PPO (Schulman al. 2017).
EXPERIMENTS
We consider both bimanual manipulation tasks and multi-agent locomotion tasks, all of which re- quire synergistic behavior, as our testbed. We establish the utility of our proposed formulations by comparing to baselines that do not use any intrinsic rewards, or use alternative intrinsic reward for- mulations. We also consider ablations of our method that help us understand the different intrinsic reward formulations, and the impact of partial differentiability. In Section 4.5 we show that our approach, with minor adaptations, continues be useful in domains with more than two agents.
4.1 EXPERIMENTAL SETUP
We consider four bimanual manipulation tasks: bottle opening, ball pickup, corkscrew rotating, and bar pickup. These environments are suggested as bimanual manipulation tasks by CCitnis et al. (2019). Furthermore, we consider two multi-agent locomotion tasks: ant push (inspired by the domain considered by Nachum et al. (2019) and soccer (adapted from the implementation provided alongside Liu et al. (2019)). All tasks involve sparse rewards, and require effective use of both agents to be solved. We simulate all tasks in MuJoCo (Todorov al. 2012). Now, we describe the tasks, state representations, and action spaces.
Environments.
The four manipulation tasks are set up with 2 Sawyer arms opposite ends of a table, and an object placed on the table surface. Two of these tasks are visualized in Figure alongside the two multi-agent locomotion tasks.
5

---

Published as a conference paper at ICLR 2020
Figure 2: Screenshots of two of our manipulation tasks and our locomotion tasks. From left to right: corkscrew rotating, bar pickup, push, soccer. These tasks all designed require agents. We learn policies for these tasks given only sparse binary rewards, by encouraging synergistic behavior via intrinsic motivation.
Bottle Opening: The goal is rotate cuboidal bottle cap, relative cuboidal bottle base, by 90" The bottle is modeled as two cuboids on top of one another, connected via a hinge joint, such that in the absence of opposing torques, both cuboids rotate together. We vary the location and size of the bottle across episodes.
Ball Pickup: The goal is lift a slippery ball by 25cm. The ball slips out when a single arm tries to lift it. We vary the location and coefficient of friction of the ball across episodes.
Corkscrew Rotating: The goal is to rotate a corkserew relative to base by 180" The corkscrew is modeled as handle attached to a base via hinge joint, such that in the absence of opposing torques, both rotate together. We vary the location and size of the corkscrew across episodes. Bar Pickup: The goal is to lift long heavy bar by 25cm. The bar is too heavy to be lifted by single arm. We vary the location and density of the bar across episodes.
Ant Push: Two ants and large block placed in an environment. The goal is for the ants to move the block to a particular region. To control the block precisely. the ants need to push together, they will often topple over when trying to push the block by themselves.
Soccer: Two soccer-playing agents and a soccer ball are placed in an environment. The goal is for the ball to be kicked into particular region, after having been in the possession of each agent for any amount of time. Therefore, the agents must both contribute to the movement of the ball.
See Section 4.5 for results on three-agent versions of the Ant Push and Soccer environments.
State Representation. The internal state of each agent consists of proprioceptive features: joint positions, joint velocities, and (for manipulation tasks) the end effector pose. The environment state consists of the current timestep, geometry information for the object, and the object pose. We use simple Euclidean metric over the state space. All forward models predict the change in the object's world frame pose, via an additive offset for the 3D position and Hamilton product for the orientation quaternion. The orientation not tracked in the soccer task.
Action Space. To facilitate learning within these environments, we provide the system with a dis- crete library of generic skills, each parameterized by some (learned) continuous parameters. There- fore, our stochastic policy maps state to 1) distribution over skills for agent A to use, 2) distribution over skills for agent to use, 3) means and variances of independent Gaussian distri- butions for every continuous parameter of skills for A, and 4) means and variances of independent Gaussian distributions for every continuous parameter of skills for B. These skills can either be hand-designed (Wolfe et al. 2010; Srriaattvaa et al. (0144) learned from demonstration (Kroemer et al. 2013); as this is not the focus of our paper, we opt to simply hand-design them. While execut- ing skill, if the agents are about collide with each other, we attempt bring them back the states they were in before execution. For manipulation tasks, if we cannot find an inverse kinematics solution for achieving a skill, it is not executed, though it still consumes timestep. In either of these cases, the reward is O. See Appendix C for more details on these environments.
4.2 IMPLEMENTATION DETAILS
Network Architecture, All forward models and the policy are 4-aaer fully connected neural net- works with 64-umit hidden layers, ReLU activations, and multi-headed output to capture both the actor and the critic. Bimanual manipulation tasks are built on the Surreal Robotics Suite (Fan et al. 2018). For all tasks, training is parallelized across 50 workers.

---

Published as a conference paper at ICLR 2020
Task Success Rate: Bottle
Task Success Rate: Ball
Task Success Rate: Ant Push
1.0
1.0
0.8
0.8
0.8
0.6
0.6
Success
0.6
0.4
0.4
0.21
0.2
0.2
0.0
10000
20000
30000
20000
40000
60000 80000 100000
0
20000
40000
60000
8000O 100000
A MujaCo Steps
MujoCo Steps
MujoCo Steps
1.0
Task Success Rate: Corkscrew
1.0
Task Success Rate: Bar
Task Success Rate: Soccer
Task Saccoss Rate
Task Succesg Rana
Rate
0.8
0.6
0.5-
0.6
0.2
0.2
0.2
20000
30000
40000
20000
50000
50000
150000 200000 MujoCo Steps
MujoCo Steps
MujoCo Steps
Random policy
Extrinsic reward only
Ours: extrinsic rieniue
aaparate-agent surprise
Extrinsic with non-synergistic surprise
Ours: extrinsic with pitrima
Figure 3: Learning curves for each our environments. Each curve depicts an average across random soeds, with standard deviations shaded. We see that is much more sample-efficient to shape the reward via ri iitiiss or rltirinsic than to rely only on the extrinsic, sparse reward signal. Also, typical formulations of intrinsic motivation surprise do not work well for synergistic tasks because they encourage the system affect the environment in ways it cannot currently predict, while our approach encourages the system affect the environment ways neither agent would if acting on its own, which a useful bias for learning synergistic behavior.
Training Details. Our proposed synergistic intrinsic rewards rely forward models fA, fa and fioint We pretrain the single-agent model f (resp. fd) on 10 samples of experience with a random policy of only agent (resp. B) acting. Note that this pretraining does use any extrinsic reward, and therefore the number of steps under the extrinsic reward is comparable across all the approaches. The joint model fjoint and policy Ty start from scratch, and are optimized concurrently. We set the trade-off coefficient A = 10 (see Appendix D We use the stable baselines (Hill et al., 2018) implementation of PPO (Schulman et al. 2017) as our policy gradient algorithm. We use clipping parameter 0.2, entropy loss coefficient 0.01, value loss function coefficient 0.5, gradient clip threshold 0.5, number of steps 10, number of minibatches per update 4, number of optimization epochs per update 4, and Adam (Kingma & Ba 2015) with learning rate 0.001.
4.3
BASELINES
Random policy: We randomly choose a skill and parameterization for each agent, at every step. This baseline serves as sanity check to ensure that our use of skills does not trivialize the tasks. Separate-agent surprise: This baseline simultaneously executes two independent single-agent cu- riosity policies that pretrained maximize the "surprise" rewards (s, genv and
Iff (s, a) 1-55h genv|] respectively.
Extrinsic reward only: This bascline uses only extrinsic sparse rewards extrinsic without shaping. Non-synergistic surprise: We learn a joint two-agent policy to optimize for the extrinsic reward and the joint surprise: full innrinnssc
non-synergisii X extrinsic This encourages curiosity-dre driven skill discovery but does not explicitly encourage synergistic aueti behavior.
4.4 RESULTS AND DISCUSSION
Figure shows task success rates as function of the number of interaction samples for the different methods on each environment. We plot average success rate over random seeds using solid lines, and shade standard deviations. Now, summarize our three key takeaways.

---

Published as a conference paper at ICLR 2020
1.0
Task Success Rate: Bottle
10
Task Success Rate: Ball
Tosk Success Rate: Corkscrew
1.5
Task Success Rate: Bar
Teak Succemn Tuli
0.8
1eah
22
02
0.2
10000
20000 40066 00000 80000 100000
40000
# MM Sugp
MagoCo Staps
MigaCe Sopp
MapoCo Stops
Baseline, pretraining
Baseline pretrained Se4 samples
extrinsic with 12
Baselina pretrained on 204 samples
Baseline protrained on 1c5 sampies
1.5
Tast Success Rate: Bottle
1.0
Task Success Rate: Bal
1.0
Task Success Rate: Corkscrew
1.0
Task Success Rate: Bar
04
2.4
24
Teak
0.
30000
106000
Hugoto Sraps
PRgCC2 Steps
MapaCa Staps
Ouri: extrinsic with pimree
Ours: extrinsic with goree
Ablation: extrinsic with growr, no analytical gradients
Figure 4: Top: Non-synergistic surprise baseline with varying amounts of pretraining for the joint model fioin. Wc see that pretraining this joint model does not yield much improvement in performance, and remains signif- icantly worse than our method (brown curve). This is sensible since the baseline does not explicitly encourage synergistic behavior, Intrinak only performs better than ri efficiency. 12
do. Bottom: Ablation showing the impact of using analytical gradients on sample Iurmss when leveraging the partial differentiability.
1) Synergistic intrinsic rewards boost sample efficiency. The tasks we consider are hard and our use of parameterized skills does not trivialize the tasks. Furthermore, these tasks require coordi- nation among the two agents, and so Separate-agent surprise policies do not perform well. Given enough training samples, Extrinsic reward only policies start to perform decently well. However, our use of synergistic intrinsic rewards to shape the extrinsic rewards from the environment accelerates learning, solving the task consistently with up to 5x fewer samples in some cases.
2) Synergistic intrinsic rewards perform better than non-synergistic intrinsic rewards. Policies that use our synergistic intrinsic rewards also work better than the Non-synergistic surprise baseline. This is primarily because the baseline policies learn to exploit the joint model rather than to behave synergistically. This also explains why Non-synergee synerrgiiii surprise used together with extrinsic reward hurts task performance (green vs. red curve in Figure 3 Past experiments with such surprise models have largely been limited to games, where progress is correlated with continued exploration (Burda et al. 2019); solving robotic tasks often involves more than just surprise- driven exploration. Figure (top) gives additional results showing that our method's competitive advantage over this baseline persists even if we allow the baseline additional interactions pretrain the joint prediction model fjoint without using any extrinsic reward (similar to our method's 's pretraining for foomposed).
3) Analytical gradients boost sample efficiency. In going from r1 intrinsic
(compositional prediction error) to riiurrisse (prediction disparity). we changed two things: 1) the reward function and 2) how it optimized (we used Equation| I to leverage the partial differentiability of f2 -_intiinic). We conduct an ablation to disentangle the impact of these two changes. Figure 4 (bottom) presents learning curves intrinsic without analytical gradients. situated in comparison to the previously shown results. When we factor out the difference due to optimization and compare r'intrinsic and r-jntinsic as different intrinsic reward formulations, r
for using r2
inttrinsic performs better than r rijutrisss (purple vs. yellow curve). This is intrinsic requires training an extra model ficint concurrently with the policy, which
expected because riz at best could match the true genv, Leveraging the analytical gradients, though, affords r intrinsic more sample-efficient optimization (brown vs. purple curve), making it better overall choice.
We have also tried using our formulation of intrinsic motivation without extrinsic reward (A = 0); qualitatively, the agents learn to act synergistically, but in ways that do not solve the "task," which is sensible since the task is unknown to the agents. See the project webpage for videos of these results. Furthermore, in Appendix D we provide a plot of policy performance versus various settings of X.
8

---

Published as a conference paper at ICLR 2020
4.5 EXTENSION: MORE THAN TWO AGENTS
It is possible to extend our formulation and proposed approach to more than two agents. Without loss of generality, suppose there are three agents A. B, and C. The only major change is in the way that we should compute the compositional predic- tion: instead of fcomposed (s,
ff s aa)), ), s".a"), we use fcomposed (s, a) One issue is that as the number of agents increases, the
(f ( s ordering of the application of single-agent forward models within fcomposed becomes increasingly important. To address this, we also tried evaluating fromposed as an average across the predictions given by all six possible orderings of application, but we did not find this to make much difference in the results. We leave a thorough treatment of this important question to future work.
), ",s,"a7 "a),,",,"""
We tested this approach on three-agent versions of the ant push and soccer environments, and found that it continues to provide useful bias. See Figure 5 In three-agent ant push, we give harder goal regions for the ants to push the blocks to than in two-agent ant push; these regions were chosen by hand to make all three ants be required coordinate to solve these tasks, rather than just two as before. In ahree-agent soccer, all three agents must have possessed the ball before the goal is scored.
Task Success Rate: Three-Agent Ant Push
Tosk Success Rate
0.8
0.6
0.4-
0.2-
0
100000
200000
300000
MujoCo Stops
Three-Apent
Tosk Suc00s Rate
Random policy
Extrinsic reward only
Ours: extrinsic with gptroe
Separate-agen agent surprise
Extrinsic with syneergggii surprise
Ours: extrinsic with gatronc
Figure 5: Left: Screenshots of three-agent versions of ant push and soccer environments. Right: Learning curves these environments. Each curve depicts an average across random seeds, with standard deviations shaded. In these three-agent environments, taking random actions almost never leads to success due to the exponentially lower likelihond of finding a valid sequence of joint actions leading to the goal, and so using only extrinsic reward does not perform well. It is apparent that proposed bias toward synergistic behavior is useful form of intrinsic motivation for guiding exploration in these environments as well.
5 CONCLUSION
In this work, we presented a formulation of intrinsic motivation that encourages synergistic behavior, and allows efficiently learning sparse-reward tasks such as bimanual manipulation and multi-agent locomotion. We observed significant benefits compared to non-synergistic forms of intrinsic mo- tivation. Our formulation relied on encouraging actions whose effects would not be achieved by individual agents acting in isolation. It would be beneficial to extend this notion further, and explic- itly encourage action sequences, not just individual actions, whose effects would not be achieved by individual agents. Furthermore, while our intrinsic reward encouraged synergistic behavior in the single policy being learned, it would be interesting to extend it to learn a diverse set of policies, and thereby discover a broad set of synergistic skills over the course of training. Finally, it would be good to extend the domains to involve more complicated object types, such as asymmetric or deformable ones; especially for deformable objects, engineering better state representations is crucial.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
Rohan is supported by an NSF Graduate Research Fellowship. Any opinions, findings, and conclu- sions expressed in this material are the authors' and need not reflect the views of our sponsors.
REFERENCES
Andrew Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learn- ing in natural and artificial systems, pp. 17-47. Springer, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.
Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In ICLR, 2019.
David Carmel and Shaul Markovitch. Exploration strategies for model-based learning in multi-agent systems: Exploration strategies. Autonomous Agents and Multi-agent systems, 1999.
Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Efficient bimanual manipu- lation using learned task schemas. arXiv preprint 1xXxv:199911374 2019.
Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-based control. In ICLR, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In ICLR, 2019.
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng. Anchit Gupta, Joan Creus-Costa, Silvio Savarese, and Li Fei-Fei. SURREAL: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference Robot Learning, 2018.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaminng awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. pp. 122-130, International Founda- tion for Autonomous Agents and Multiagent Systems, 2018
Elena Gribovskaya and Aude Billard. Combining dynamical systems control and programming by demonstration for teaching discrete bimanual coordination tasks to a humanoid robot. In 2008 3rd ACM/EEE International Conference on Human-Robor Interaction (HRI), pp- 33-40 IEFE, 2008.
Shixiang Gu, Ethan Holly. Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Nick Haber, Damian Mrowca, Li Fei-Fei, and Daniel LK Yamins. Emergence of structured behav- iors from curiosity-based intrinsic motivation. arXiv preprint
2018.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable baselines. 2018.
stable-baselin
Ping Hsu. Coordinated control of multiple manipulator systems. IEEE Transactions on Robotics and Automation, 9(4)400-410, 1993.
Sandy H Huang, Martina Zambelli. Jackie Kay, Murilo F Martins, Yuval Tassa, Patrick M Pilarski, and Raia Hadsell. Learning gentle object manipulation with curiosity-driven deep reinforcement learning. arXiv preprint 1rX3vv1903.088: 2019.
10

---

Published as a conference paper at ICLR 2020
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A Ortega. DJ Strouse, Joel Leibo, and Nando de Freitas. Intrinsic social motivation via causal influence multi-agent RL. In ICML, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Oliver Kroemer, Christian Daniel, Gerhard Neumann, Herke Van Hoof, and Jan Peters. Towards learning hierarchical skills for multi-phase manipulation tasks. In 2015 IEEE International Con- ference Robotics and Automation (ICRA), pp. 1503-1510. IEEE, 2015.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-133 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, In ICLR, 2016.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emer- gent coordination through competition. ICLR, 2019.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipula- tion via locomotion using hierarchical sim2real. arXiv preprint 1rX.vv:908.00522. 2019.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine. Neural network dynamics for model-basee deep reinforcement learning with model-free fine-tuning. In 2018 IEEE Interna- fional Conference Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Pierre-Yve Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):265-286, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In ICML, 2019.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994.
Mare H Raibert, John J Craig, al. Hybrid position/force control of manipulators. Journal of Dynamic Systems, Measurement, and Control, 103(2):126-133, 1981.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Christian Smith, Yiannis Karayiannidis, Lazaros Nalpantidis, Xavi Gratal, Peng Qi, Dimos V Di- marogonas, and Danica Kragic. Dual arm manipulationa survey. Robotics and Autonomous systems, 13410331300 2012.

---

Published as a conference paper at ICLR 2020
Siddharth Srivastava, Eugene Fang. Lorenzo Riano, Rohan Chitmis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible ceinnerr-ddeeeddenn interface layer. In 2014 IEEE international conference robotics and automation (ICRA), pp. 639-646.
IEEE, 2014.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.008144 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ronald J Williams. Simple statistical gradient-folloingg algorithms for connectionist reinforcement learning. Machine learning, (444)2229-266 1992.
Jason Wolfe, Bhaskara Marthi, and Stuart Russell. Combined task and motion planning for mobile manipulation. In Twentieth International Conference on Automated Planning and Scheduling, 2010.
Ning Xi, Tzyh-Jong Tarn, and Antal K Bejezy. Intelligent planning and control for multirobot coordination: An event-based approach. IEEE transactions robotics and automation, 12(3): 439-452, 1996.
Zollner, Tamim Asfour, and Rüdiger Dillmann. Programming by demonstration: Dual-arm ma- nipulation tasks for humanoid robots. In 2004 IEEE/RSJ International Conference Intelligent Robots and Systems (TRES)IEEE Cat. No. 04CH37566), volume 1, pp. 479-484, IEEE, 2004.
12

---

Published as a conference paper at ICLR 2020
A
PSEUDOCODE
Here full pseudocode of our training algorithm described in Section 3.3
Algorithm PRAAN-SSWWRRISTTTCPLLLCCC......
Input:
иo, an initial policy.
Input: M, an MDP for synergistic task.
Input: the number of episodes of data with which to train single-agent models.
Input: a, step size.
for i 1, do
Append episode experience MM with only agent A acting to data buffer DA Append episode of experience in M with only agent B acting to data buffer D Fit forward models fAA f to predict next states in DA, DB // Pretrained & fixed. // Data for joint model, only needed if using intrinnie while "e has not converged
Dioint <-0 0
10
batch of experience tuples (St, d2y rtt restrimsi s++1) from running He in M.
11
12
if using r2
runtrinsic then
Append D to Dioint and fit forward model fiuin to predict next states in Disint
for (st, ate retrinsii S241) E D do
Replace pextrissic with y fflll az, $1+1). // Shape reward, see Section 3
15
V (0) + POLICYGRRDIEENT(TO, D) intrinsic then
16
17
if using r
Update (0) with analytical gradients per Equation [
18
0 - 0 (V)
Or Adam (Kingma
2015
B DERIVATION OF EQUATION
When using r2 intrinsic the objective to optimized can be written as:
((0) Er (T) E, Σ full (st,
(st, ac,
E
r2
intrinsic
(st, ae, + X x extrinsic (st)
1=U
We will write (0) in particular way. Let Fr (sp, ao, 81, ,1,a1, mrg St) be random variable denoting trajectories up to timestep I, but excluding at- We have:
VoJ(O) VoE, (T)
E VoE,, (ar Ba (sc. a1, -1],
t=
where we have used the fact that trajectories up timestep / have no dependence on the future 8141, at+l. Sr, and we have split up the expectation. Now, observe that the inner expectation, Ba [i (r (st, a),, is dependent on 0 since the a, sampled from the policy тæ; intuitively, this expression represents the expected reward of with respect to the stochasticity in the current policy. To make this dependence explicit, let us define rfoll((s) (se) F (rrfull ae, )). Then:
VoJ(O)=
V EgE, [r (st)
t=
Σ
( dr.
Po (p.
full (8t) Vo log po (o (st) di,
T
Er, [r Jull (8)) Va log + E,, (s)))
13

---

Published as a conference paper at ICLR 2020
where in the second line, used both the product rule and the REINFORCE trick (Williams 1992).
Now, let Tj = (5o, ao, 81, a1, .... St, ai) denote trajectories up to timestep t, including as (unlike Tz). Putting back Ba(=zo(ss) (nfull (5t, -): ) in place of r (st) gives Equation
7
VoJ(O)
Σ
E,, Ba (st (st. ae, V1 log + E,, Vo mo(a)) (st at, I)
I=0
Σ t=0
E,, (r [st at, Vo E, Vo Vo iuriissi (st, oe; )]]-
U
In the second line, have used the facts that Ty and the extrinsic sparse reward do not depend d,- Note that we can estimate the term E,, [V (na))) [rd (St, Oz, of trajectory data T! T"..,"", for any timestep t. T",
-t)-)]] -)] empirically using a batch
C ADDITIONAL ENVIRONMENT DETAILS
C.I MANIPULATION TASKS
We provide additional details about the action space of each manipulation environment.
The following table descrihes the parameterization of each skill in the library, as well as which environments are allowed utilize each skill:
Skill Environments
Continuous Parameters
top grasp bar, ball, bottle
end effector position, end effector z-orientation
side grasp
bottle, corkscrew
end effector position, approach angle
go-to pose
ball, corkscrew
end effector position, end effector orientation
lift
bar, ball
vertical distance to lift end effector none (wrist rotates at current end effector pose)
twist bottle
rotate corkscrew
rotation axis, rotation radius none
no-op
all
The following table describes the search space of each continuous parameter. Since the object pose known in simulation, are able leverage designing these search spaces:
Continuous Parameter
Environments
Relevant Skills
Search Space
end effector position (unitless)
bar
tup grasp
[-1, I] interpolated position along bar
end effector position (meters)
ball, bottle. corkscrew grasps. go-to pose
[-0.1. 0.1] x/y/z offiett from object center
end effector z-orientation
bar, ball, bottle top grasp
[0, 2x]
approach angle
bottle, corkscrew side grasp
end effector oricntation
ball, corkscrew
go-to posc
[0,2a] r/ply Euler angles converted quat
distance to lift (meters)
bar, bottle
lift
[0, 0.5)
rotation axis
corkscrew
rotate
[[-0.1, 0.1] xfy offiet from object center; vertical
rotation radius (meters)
corkscrew
rotate
0.2
Note that inverse kinematics feasibility checks allow the system to learn to rule out end effector poses which are impossible to reach, since these cause no change in the state other than consuming timestep, and generate 0 reward.
14

---

Published as a conference paper at ICLR 2020
C.2 LOCOMOTION TASKS
We provide additional details about the action space of the locomotion environments. For both the push and soccer tasks, We follow Nachum al. (2019) and pre-train four skills: moving up. down, left, and right on the plane. Each skill has one continuous parameter specifying an amount to move. So, at each timestep. the policy must select both which direction move and how much to move in that direction. All training hyperparameters are unchanged from the manipulation tasks.
C.3 POLICY ARCHITECTURE
Figure shows diagram of our policy architecture.
logits for
sample
logits
State
15g
action
Ju,
environment
pi. at for skills
sample
categorical distribution over skills for A, calegorical distri- bution over skills B, 3) means and variances of independent Gaussian distributions for every continuous parameter of skills for A, and 4) means and variances of independent Gaussian distributions for every contin- vous paramcter of skills for B. To sample from the policy, we first sample skills for A and B, then sample all necessary continuous parameters for the chosen skills from the Gaussian distributions. Altogether, the two skills two sets parameters form action, which can be fed into forward models for prediction.
Figure 6: The policy Th maps
D IMPACT OF COEFFICIENT
We conducted experiment study the impact of the trade-off coefficient the performance of the learned policy. When À = O. no extrinsic reward is used, so the agents learn to act synergistically, but ways that do solve the "task," which sensible since the task unknown them. Our experiments reported in the main text used À = 10. See Figure for the results of this experiment.
Impact Task Success Rate
Task Success Itaee
0.5
0.4
0.2
0.0
Figure 7: Task success rate of learned policy at convergence for the bar pickup task, using our best best-performing reward function rz inrimic Each result is averaged across random seeds, with standard deviations shown. We infer that once reaches high enough value for the extrinsic rewards to outscale the intrinsie rewards when encountered, the agents will be driven toward behavior that yields extrinsic rewards. These extrinsic, sparse rewards are only provided when the task is successfully completed.
15


Reviews:
[{"review_id": "SJxFBZDpYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on using intrinsic motivation to improve the exploration process of reinforcement learning agents in tasks with sparse-reward and that require multi-agent to achieve. The authors proposed to encourage the agents toward the actions which changed the world in the ways that \"would not be achieved if the agents were acting alone\". The experiments are done with dual-arm manipulation.\n\nThe idea of guiding the agents toward the actions that they cannot do without concurrent cooperation is interesting. In this paper, it is presented by two types of intrinsic rewards: compositional prediction error and prediction disparity. The core component is the composition of these single-agent prediction model (f^{composed}). Although the formulation proposed is only based on intuition, the authors did enough ablation study to highlight the advantage of this loss function. \n\nAreas to improve:\n+ The objects used in the experiment are symmetric, it is good to open your study to the task in which the objects are asymmetric or even deformable.\n+ It is good to extend to the problem of multiple-agents (>2), while the order of agents who acts is important to compute \"the expected outcome with individual agents acting sequentially\" (for now, you only assume that the A will act first then the B acts later). \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeoB2EbcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel algorithm for encouraging synergistic behavior in multi-agent setups with an intrinsic reward that promotes the agents to work together to achieve states that they cannot achieve individually without cooperation. The paper focuses on a two-agent environment where an approximate forward dynamics model is learnt for each agent, and can be composed sequentially to predict the next environment state given each agent’s action. However, this prediction will be inaccurate if the agent’s affected the environment state in such a way that individual dynamics model cannot predict i.e. synergistic behavior was produced. This prediction error is used as extrinsic reward by the proposed approach, while also having a variant where the true next state is replaced by another approximation of a joint forward model which allows for differentiability of actions with respect to the intrinsic reward. Empirical analysis shows that this intrinsic reward promotes synergetic behavior on two-agent robotic manipulation tasks and achieves better performance that baselines and ablations.\n\nI vote for weak accept as the paper proposes a novel intrinsic reward for promoting synergetic behavior in multi-agent systems, while also demonstrating that such an intrinsic reward can be differentiable if a joint forward dynamics model is approximated in addition to individual forward dynamics models given each agent’s actions. The paper does not show experiments beyond 2 agents and the four robotic manipulation tasks have been shown to work when provided with generic skills as an action space, which requires hand-defining or learning by demonstration.\n\nFrom the description of the random policy, it is stated that a random policy over skills serves as a sanity check to ensure that the skills do not trivialize the task. This seems to suggest that the extrinsic reward only baseline did not use these skills and was disadvantaged. Some clarification is required here - did the extrinsic reward only baseline use the same skills as the proposed method? If it did, it would obviate the need to have a random policy sanity check. \n\nThe paper suggests a baseline for separate arm surprise. In a similar vein, why wasn’t a joint-arm surprise baseline employed, which can basically treat both arms as a single agent?\n\nSynergistic behavior seems hard to achieve for a large number of agents, but the paper does not give insights into whether such an algorithm will work for more than 2 agents. Typically multi-agent systems in prior work have worked with a large number of agents in environments other than robotic manipulation - such experiments may help in strengthening the proposed method.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlLBP6CKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is technically sound and easy to read. I very much welcome that.\n\nThe authors address the important issue of exploration in reinforcement learning. In this case, they propose to use reward shaping to encourage joint-actions whose outcomes deviate from the sequential counterpart. Although the proposed intrinsic reward is targeted at a particular family of two-agent robotic tasks, one can imagine generalizing some of the ideas here to other multi-agent learning tasks.\n\nThe authors did mention testing with lambda=0 (intrinsic only), which did not aim to solve the 'task' but would be informative in terms of understanding the results of this particular bias. It would be interesting to include a plot on the effect of lambda across a range, say from 0 to 10.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the important issue of exploration in reinforcement learning. In this case, they propose to use reward shaping to encourage joint-actions whose outcomes deviate from the sequential counterpart. Although the proposed intrinsic reward is targeted at a particular family of two-agent robotic tasks, one can imagine generalizing some of the ideas here to other multi-agent learning tasks.

The reviewers agree that the paper is of interest to the ICLR audience.
Decision: Accept (Poster)