Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation
Authors: Cansu Sancaktar
Sebastian Blaes
Georg Martius
Venue: NeurIPS 2022 Accept (2022)
Keywords: Intrinsic Motivation
Reinforcement Learning
Model-based Planning
Manipulation
Zero-shot Generalization

Abstract: It has been a long-standing dream to design artificial agents that explore their environment efficiently via intrinsic motivation, similar to how children perform curious free play. Despite recent advances in intrinsically motivated reinforcement learning (RL), sample-efficient exploration in object manipulation scenarios remains a significant challenge as most of the relevant information lies in the sparse agent-object and object-object interactions. In this paper, we propose to use structured world models to incorporate relational inductive biases in the control loop to achieve sample-efficient and interaction-rich exploration in compositional multi-object environments. By planning for future novelty inside structured world models, our method generates free-play behavior that starts to interact with objects early on and develops more complex behavior over time. Instead of using models only to compute intrinsic rewards, as commonly done, our method showcases that the self-reinforcing cycle between good models and good exploration also opens up another avenue: zero-shot generalization to downstream tasks via model-based planning. After the entirely intrinsic task-agnostic exploration phase, our method solves challenging downstream tasks such as stacking, flipping, pick & place, and throwing that generalizes to unseen numbers and arrangements of objects without any additional training.

Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation
Cansu Sancaktar
Sebastian Blaes
Georg Martius
Max Planck Institute for Intelligent Systems Tübingen, Germany
sancakaa,, sebastian. blaes, georg. martius]@tue. mpg. de
Abstract
It has been a long-standing dream to design artificial agents that explore their environment efficiently via intrinsic motivation, similar to how children perform curious free play. Despite recent advances in intrinsically motivated reinforce- ment learning (RL), sample-efficient exploration in object manipulation scenarios remains a significant challenge as most the relevant information lies in the sparse agent-object and object-object interactions. In this paper, we propose structured world models to incorporate relational inductive biases in the control loop to achieve sample-efficient and interaction-rich exploration compositional multi-object environments, By planning for future novelty inside structured world models, our method generates free-play behavior that starts to interact with objects early on and develops more complex behavior over time. Instead of using models only to compute intrinsic rewards, as commonly done, our method showcases that the self-reinforcing cycle between good models and good exploration also opens up another avenue: zero-shot generalization to downstream tasks via model-base planning. After the entirely intrinsic task-agnostic exploration phase, our method solves challenging downstream tasks such as stacking, flipping, pick & place, and throwing and generalizes unseen numbers and arrangements of objects without any additional training."
Introduction
Curious free-play has been identified as a driving force in child development, allowing children to efficiently explore their environment and build an understanding of the world [1]. Such intrinsically motivated exploration schemes are especially attractive in open-ended learning scenarios to guide an agent even without extrinsic tasks and corresponding rewards. Similar to how children learn, we want Reinforcement Learning (RL) agents to learn through play and then be able solve new tasks quickly. We address both challenges in this work.
Minimization novelty (or surprise) is prominent formulation of curiosity, with several psycho- logical studies showcasing the role of novelty in children's curious exploration [2-4]. Novelty as an intrinsic reward signal has also been adopted RL, where agents try resolve cognitive disequilibrium [5, 6]. However, applying entirely intrinsic task-agnostic exploration to object manip- ulation scenarios
sample-efficient manner an ongoing challenge as the relevant information lies in the sparse agent-object and object-object interactions. These compositional multi-object manipulation environments highlight one of the significant weaknesses of the current novelty-based intrinsic motivation methods: a novel stimulus alone does not necessarily mean that it contains useful
Code and videos are available at https::/martiuu-titt github. io/cee-us
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Intrinsic
Extrinsic Phase: Zero-sbot Generalization
Planning imagination with intrinsi rewards
Model-based Planning
Observation
Eita
Stacking
Pick & Place
Forward
Exsemble of CNNNS
2611 fv. [d,,
Generate data with model
Update model with data
Maximize ensemble disagreement
Tarowing
Flipping
Figure Intrinsic motivation for epistemic uncertainty reduction with structured world models yields capable models for zero-shot task planning. In an intrinsic phase, the system plans for maximum epistemic uncertainty with an ensemble of GNNs and uses the actively collected data to update the model. This iterative procedure ultimately reduces the overall epistemic uncertainty of the system, i.e. maximizing information gain. The learned models can be used to solve complex manipulation tasks using online planning.
or generalizable information for an individual [7]. Thus, curious exploration needs attend subset of possibilities, as supported by studies in psychology [8, 9]. Analyzing children during free play accumulated evidence that infants have innate biases and heuristics help guide their attention toward relevant and informative features of the environment [10, 11]. Our goal is to improve curious exploration in RL by incorporating appropriate inductive biases. We hypothesize that viewing the world as a collection of entities and their interactions is such "useful" inductive bias, which was also put forward Tsividis al. [12]. However, instead of utilizing explicit theory-based modeling, we exploit a relationa inductive bias by using Graph Neural Networks (GNN) as our choice of model [13] and learn everything from interactions. We postulate that curious exploration via epistemic uncertainty of such model leads to the collection of valuable data it inherits this structure.
This paper shows how online planning methods can efficiently exploit learned models. both for explo- ration and zero-shot generalization to tasks. Recent advances in ceneral-purpose model predictive control methods have reduced their complexity and show strong performance when good forward models are available [14, 15]. We build on this work to create self-rein self-reinforcing cycle between learning good models and good exploration. Thanks to their ability plan for new task without further training, models efficiently track the naturally changing exploration targets and perform downstream tasks a zero-shot generalization manner.
We propose CEE-US: Curious Exploration using Epistemic Uncertainty via Struetured Models that achieves sample-efficient and interaction-rich exploration in multi-object manipulation environments. To our knowledge, we are the first to use GNN ensemble disagreement for computing intrinsic motivation signals in RL. A further major contribution of our method is to combine GNN-based epistemic uncertainty with planning methods in task-agnostic setting and demonstrate zero-shot generalization to challenging object manipulation tasks. We provide brief comparison related work in Table and provide more details in Sec.
Table Comparison of CEE-US with baselines in terms of differences in paradigms.
Planner
Learns dynamics
Zero-shot task generalization
Combinatorial generalization
Method
Observation type
or policy
CEE-US
Proprioceptive
Planner
MLP+ iCEM
Proprioceptive
Planner
X
Disagreement [16]
Propr. Image
Policy
X
RND [17]
Propr. Image
Policy
ICM [6]
Propr. Image
Policy
X
X
Plan2Explore [18]
Image
Policy
(offline RL)
X

---

2 Method
We focus intrinsically motivated learning to prepare for future tasks. Thus, have task-agnostic RL setting without extrinsic rewards or any other information regarding future tasks/goals during the initial play phase. Our approach trains structured world model to capture the forward dynamics of the environment from active exploration data. In particular, we are using a Graph Neural Network (GNN) where the nodes correspond to objects. We rely on model's epistemic uncertainty direct exploration during free play. Our focus is on compositional multi-object environments, where an actuated agent can independently manipulate different objects. The terms (actuated) agent and robot are used interchangeably.
2.1 Preliminaries
In this work, we consider a fully observable Markov Decision Process (MDP) setting, The MDP is given by MM= (S, A, P:y. Rggr), with the continuous Mate-space S E R the continuous action- space A E R the transition kernel Pa and the reward function R Furthermore, we consider an object-oriented state representation, i.e. the state-space factorizes into the different entities in the environment S S x S.... . x Sx X Sagent, where N denotes the number of objects.
In the typical RL setting, the agent's goal is to learn policy a T(. 8) that maximizes the future (discounted) cumulative reward G2 Lk=g 7 Reck, with the discount factor 0 < y S 1. Such policy can be learned as a neural network with RI. algorithms, or a planning method can be used to maximize the same quantity.
2.1.1 Planning and Model Predictive Control
Planning methods use a model f of the transition kernel p and a reward function R. to optimize sequence of actions on the fly. The potential advantage of planning methods is that they can optimize for different reward functions without further adaptation. For this to successful, good transition model / needs to be learned, and the reward function needs to be known or discovered. More formally, for fixed planning horizon H, the action sequence a = (a=+h)==d maximize
H-1 is optimized to
H-1 Σ
a arg max
where Sc4-h are imagined states visited by rolling out the actions using S. which is assumed to be deterministic. For this procedure to optimize the same infinite horizon reward with discounting, ? and value function V($++) could be added to Eq. 1, sacrificing the flexibility in exchanging It
We use zero-order trajectory optimization find actions according to Eq. 1 and model predictive control (MPC) to convert the open-loop planning policy into closed-loop policy (re-planning after every step in the environment). In particular, we use the improved Cross-Entropy Method (iCEM) [14], which was recently proposed as sample efficient trajectory optimizer.
2.2 World Model with Graph Neural Networks
As the forward dynamics ʃ S x A + S (world model) is an integral part of the planning procedure, it needs to be able to capture the true dynamics well. We employ an ensemble of message-passing GNNs [13]. Each object corresponds to node in the graph, and the node attributes {oî E S, n = 1, NN are given by an object's features such as position, orientation, and velocity timestep t. The state representation of the actuated agent "gent E Sugent similarly contains position and velocity information about the robot.
The agent representation is differentiated from the object nodes as it has a direct cause-and -adue--nndfffer relationship with the actions. The agent's state and the generic action are represented as global context The GNN is fully connected, i.e. there is an edge between all nodes. In GNNs, the node update function Gnode models the dynamics of individual entities, and the edge update function Gedge captures their pairwise interactions.
3

---

Moving one object
Lifting object
Stacking objects in hand
Throwing objects out reach
Hlipping objects at once
Moving object(s) by roiling thren
Transitions
Emergent behaviors observed during free play with CEE-US in CONSTRUCTION, Along the time axis, 2000 transitions correspond one training iteration of CEE-US, with 20 rollouts (episodes) of length 100. After only 30 iterations of free play, the robot starts lifting objects.
Figure 2:
The node attribute computation for the objects is given by:
[st
sgeee ar)
(i.j) = Jedge ( ([s), sl,
(3)
sf ++1 = Jnude [[s), C, agg (c )))
(4)
where ......] denotes concatenation, et (+-) is the edge attribute between two neighboring nodes (i,j), and is the global context information. The inpuuutttioninvariinn aggregation function given by aggr. The Gnode and Sedge are both Multilayer Perceptrons (MLP).
The context update, i.e. the transition of the agent's state, is computed using the global aggregation of all edges similar to the global context formalism in Battaglia et al. [13]:
8 Iglobal ([c, aggri, (ce ()))
(5) where gglobal denotes the global node MLP. By providing all information to the prediction of the agent, we ensure an accurate modeling of agent-object interactions and their influence on the agent, which is paramount in object manipulation environments. Using this representation, found that single message passing step is sufficient, thus yielding fast inference times.
Moreover, to make the model focus capturing the changes accurately, let the GNN predict A&;11 = 8t+1 - St, instead the absolute next state, such that f(s, a) = s + GNN((s.a)
2.3 Epistemic Uncertainty as Intrinsic Reward
We train an ensemble of GNNs ((om) (Fig. 1), where M denotes the ensemble size. The epistemic uncertainty, i.e. the uncertainty due to lack of data, can be approximated by the disagreement of the ensemble members' predictions measured by the trace of the covariance matrix [15]:
(1 .). = (Cov (1 (em a:) m 1...,,,,,,))))) M))).
(6)
Connections Information Gain If more data becomes available from region where the ensemble members disagree, then we expect the disagreement in this region to shrink, given enough capacity the predictors. By guiding the exploration to regions with large model disagreement or high epistemic uncertainty, we explicitly aim for maximizing information gain [19-21]. Interestingly. as we can predict the uncertainty for unvisited states, our approach more related to predicted information gain [22].
2.4 The CEE-US Algorithm
CEE-US has two different phases: the intrinsic free-play phase (Alg. 1), where learning occurs with active exploration, and the extrinsic phase (Alg. 2), where we apply the learned model to solve downstream tasks zero-shot without any additional training.
Intrinsic Phase In each iteration, the zero-order trajectory optimizer is used to plan for action sequences with high cumulative epistemic uncertainty in the imagined trajectories of the world model. The collected rollouts are added to the buffer, and the ensemble members of the world model are

---

trained on the observed state transitions (st, az, S1+1) for fixed number of epochs minimizing the loss function:
Lm | - fem(sr (2)))l2)
for each ensemble member m with independently sampled mini-batches, where As++1 = S141 St denotes the actual change in the next state observation. The total loss function is given by: M f. Zi=o f.m- Afterwards, the process of data collection and training is repeated. The pseu- docode for the intrinsic phase is provided in Alg. 1. Lines 3-9 correspond to one training iteration.
Algorithm I CEE-US; Free Play in Intrinsic Phase
Algorithm 2 CEE-US: Zero-shot Generalization in Extrinsic Phase
1: Input: {(fe ) Randomly initialized ensemble of GNN;
with members, emply dataset, Planner: iCEM planner
Input: {) }: Ensemble GNNs with m members
with horizon F
learned during intrinsic phase, Plannar: iCEM planner with hori-
while explore
b- Explore with MPC and intrinsic reward
zon H. Rank: Reward function of extrinsic lask
3: for
eesiondde do
while not fooe do
Plan maximize task reward in model an +- Flanner(Ss: {(fo )
4: for = to do D Plan to maximize model uncertainty
3:
Rask) Rus))
& task reward
#, Planner (81, ([f9m))m m= Rt)
Fiç,
8c-1 env wwv.etep((s... oe)
6:
- env. stw.sttp(s:;02 0z)
7: D D (t--)))=)
Thain models dataset for epochs 9: Pm <- optimize Pm, using C.m on D for m 1,...,
For 1 to do
10: return {[fo m) m== }.
Extrinsic Phase The learned world model is used perform planning for several downstream tasks, which we assume to be given in terms of reward functions Rusa (8, s'). The pseudocode is presented in Alg. 2. Here, no further adaptation to the model is performed, although our method could be extended to incorporate a potential fine-tuning procedure.
3 Experiments
In our empirical evaluation, we analyze the performance of CEE-US on two different object manipu- lation environments to answer the questions: How much does the structural inductive bias introduced by GNNs help model learning and control? Does the free-play phase create rich interaction data that helps downstream task performance? Can we solve challenging manipulation tasks in zero-shot manner? The two environments we consider are:
Playground An actuated spherical agent can slide in the x-y directions and push four different object types (light cube, heavy cube, pyramid, and cylinder) along the x-y directions and also rotate them around the z-axis (Fig. 3). Each object has 3 Degrees of Freedom (DoF). Object types are uniquely identified by their color.
Fetch Pick & Place Construction This is extension of the Fetch Pick & Place environment [23] to more cubes [24] (Fig. 2). A 7-DoF robot arm is used to manipulate blocks (each with DoF). The actions F control the gripper movement in Cartesian coordinates and the gripper opening/closing. The robot state Sagent E 1100 contains the end-effector position and velocity and the gripper-state (open/close) and gripper-velocity. Each object's state s' E g22 is given by its position, orientation (in Euler angles), and linear and angular velocities. We replaced the table in front of the robot with a large plane so that objects cannot fall off during free play. However, they can still be pushed or thrown outside of the manipulability range of the robot. Originally, each object state contained the object's position relative to the gripper. We remove this privileged information from the objects' state, as it already introduces relational bias in the raw state representation.
3.1 Structured vs. Unstructured World Models
We analyze how curious exploration based on minimizing epistemic uncertainty performs when using ensembles of GNNs and ensembles of MLPs (Fig. 3). ML.Ps, with their fully connected layers and monolithic input representations, do not offer isolation of information and incorporate no explicit relational inductive bias [13], thus constituting good baseline for our structured world models. Results in PLAYGROUND show that the GNN ensemble leads much faster to interaction richh data
5

---

Environment
100%
Cyleder Pyramas Cobl
Reaaive tma
CEE-US
Louce
10%
www.
Agent postions
0.0
0%
125
250
Training iteration
-12
100%
MUP ICEM
Rolotvee time
Training iteration
Figure Structured world models lead to more object interactions. In the playground environ- ment, (left) we compare active exploration with an ensemble of GNNs (top row) vs. ensemble of MLPs (bottom row) using epistemic uncertainty as an intrinsic reward (M 5). Middle: The heatmaps show the uncertainty of the ensemble members for hypothetical agent positions over spatially discretized grid different training iterations (see Suppl. C.6). Right: The relative time (the fraction of time steps) the agent spends moving the different objects in the playground is illustrated throughout free play.
than its MLP counterpart. As visualized in Fig. 3, the uncertainty produced by the GNN ensemble already localized around objects after 25 training iterations, leading targeted agent-object and object-object interactions. For the MLP, it takes more than 100 iterations to start generating useful uncertainty estimates and therefore object-agent interactions. Also, note the pronounced uncertainty at the walls for the MLP. A more fine-graines analysis of the interaction times of CEE-US compared the baselines will be provided below. In Suppl. D, we also show how the resulting multi-step dynamics predictions with structured world models are more accurate compared to the MLPs. This is a key component for the reiffrenin cycle between good models and good exploration as better model means: the agent can plan for more complex behavior earlier and learn from these experiences faster.
3.2 Interaction-Richnes of Generated Exploration Data
We perform an analysis of the data generated during intrinsic free-play. We compare the performance of our method to the unstructured MLP+iCEM variant (CEE-US without GNNs) as well other intrinsic motivation baselines:
Disagreement [16] The one-step disagreement of an MLP ensemble used intrinsic reward to train an exploration policy.
Random Network Distillation [17] A predictor network, corresponding to the forward model, tries to match the output target network with random weights. The discrepancy between the two networks is used as an intrinsic reward (a type of state visitation count for continuous domains).
Disagreement
RND
100%
100%
100%
relative time
50%
0%
0%
50
100
200
250
Transitions x 2000
Trensitions 2000
Transitions 2000
Transitions x 2000
(a) Agent in free space
(b) Agent at wall
(c) 1 object moves
(d) 2 or more objects move
Figure Interaction rate during intrinsic exploration in the PLAYGROUND environment. The relative interaction times show that CEE-US quickly starts interacting with the objects. MLP iCEM similar but one order of magnitude less data efficient. The baselines rarely engage interesting interactions, mostly staying in free space or at the walls (three independent seeds).

---

MLP+ iCEM
Disagreement
RND
50%
40%
30%
relabive time.
25%
20%-
15%6
056
0%
50 150 zóo 250
150 200 250 300
100
300
Transitions x 2000
Transitions x 2000
Transitions 2000
Transitions x 2000
(a) 1 object moves
(b) 2 or more objects move
(c) object(s) flipped
(d) object(s) in air
Figure Rich interactions during free play in CONSTRUCTION by CEE-US. Interaction metries
of free-play exploration count the relative amount of time steps spent in moving one object (a). moving two and more objects (b), flipping object(s) (c), and moving objects into the air (d). CEE-US engages quickly in all sorts of interactions with objects. The ablation with MLPs eventually performs similarly but less sample efficient. The RL baselines perform poorly on these timescales. We used three independent seeds.
The intrinsic reward is defined the error between MLP forward model's next state prediction and the actual next state. As this method needs access to the true next state, the intrinsic reward can only be computed retrospectively.
[CM
The intrinsic reward is used to train an exploration policy in these methods. We use the implementation from Laskin et al. [25] that uses DDPG [26]. In Fig. 4, we present different metrics quantifying the amount and type of interactions that occur during the free play in the PLAYGROUND environment. It is eminent that CEE-US is only shortly interacting with the walls and then quickly interacts with one and two objects simultancously. The next best method is the ablation our method using MLP models needing about ten times more interactions, as explained in Fig. 3. The baselines mainly moving in free space and interacting with the walls. RND starts interacting with the objects at least 10% of the times-steps.
In the CONSTRUCTION environment, the situation even more drastic, as shown in Fig. 5. CEE-US starts repeatedly moving one object after only eight iterations (each 2000 environment steps), picks up objects after around 30 iterations, and continues with throwing and flipping objects and moving shown in terms
multiple objects frequently after 50 iterations. In Fig. 2 exemplary behaviors of snapshots at different stages of learning. We believe this sample efficiency is remarkable. The ablation of our method with MLP models is also able to engage in interesting and diverse interactions, but at a much slower rate (Fig. 5). The policy learning baselines rarely interact with objects, even after 300 iterations with 20 episodes each.
3.3 Zero-Shot Generalization to Downstream Tasks
We demonstrate that after the free-play phase, the learned models can be used for zero-shot solving of complex downstream tasks without the necessity to generate data perform further training. Furthermore, thanks to the combinatorial generalization capabilities GNNs, the model can be used with different number of objects in the environment than seen during free play. The only baseline we consider for the zero-shot generalization performance is MLP + +-CEM,,, as none of the other methods discussed in the previous section solve downstream tasks without additional training.
In PLAYGROUND, we consider the task of bringing all objects to a single target location, as shown in Fig. 6. The reward function is the sum of negative distances of all objects the target outside threshold distance. The target location sampled randomly at the beginning of each episode. The
Table 2: Success rates in the PLAYGROUND envi-
ronment for the downstream task Push objects to one location, with variable number of objects after 100 and 250 training iterations of free play. Computed over three random seeds, each evaluated on 100 roll- outs.
(a) (b) 1 500 Figure 6:
Downstream task in PLAY- GROUND: move all objects to place, as solved by CEE-US.
4 @10 one each
Objs@@200 one each
Objs@ 250
Objs@
random
random
CEF-US
0.70
0.91 0.06
0.93 +
0.04
MLP iCEM
0.08 4: 0.01
0.90 :: 0.03
7

---

success rates for different numbers of objects are shown in Table 2. We define success rate in the multi-objecc setup as the fraction of objects solved relative to the total number of objects spawned in the environment. After 100 training iterations of free play. CHE-US already achieves 70% success rate, whereas the MIP version only reaches 8%. After 250 iterations, both methods are on par. Since CEE-US can also deal with variable input dimension, we also consider the task with more or less objects with randomly sampled types.
In CONSTRUCTION, several challenging manip- ulation tasks need be solved: pick & place, stacking, throwing, and flipping, shown in Fig. 1. The reward functions are detailed in Suppl. C.3.1. In Fig. 7, we present the suc- cess rates in each of the tasks for CEE-US and the MI.P-based planning baseline MLP+ iCEM, where applicable. We find remarkable success rates across the board for CEE-US, even in chal- lenging tasks such as stacking and throwing. For stacking, we report success rate of 1 if the tower with all the objects in the environment is stacked and 0 otherwise. For the multi-tow task, denoted by 2 + 2, we consider success only when two towers stacked. We recommend visiting website for videos of these tasks. In Suppl. E, we present additional experiments combining RND as intrinsic reward with model- planning, similar Lambert et al. [27], in CONSTRUCTION, where look at both structured and unstructured world models. The results showcase the benefits of using the model's 's own epistemic uncertainty estimate to guide exploration, as it is the case with ensemble disagreement, leading to more accurate and more robust dynamics models and better zero-shot downstream task generalization.
1.0
success rate
0.5
0.0
2
3
5
6
2
3 2+2 2
3
4
2
3 4 5
Pick & Place Stacking Throwing
Flipping
# of objects
Figure Zero-shot performance on down- stream tasks in CONSTRUCTION for CEE-US and MLP+ + iCEM.
3.4 Offline Learning of Downstream Tasks from Exploratory Data
This experiment investigates the quality of the data collected by the different methods in the free-play phase for solving downstream tasks via offline RL. This allows us to compare to other baselines that do not support zero-shot generalization. Additionally, it is one way of obtaining task-specific policies. In offline RL., a control policy is learned from a fixed dataset, where we repurpose the exploration data. The results are shown in Table 3. Offline policy learning is performed via CQL [28]² and training details can be found in Suppl. C.5. To learn different downstream tasks from the same dataset, reward relabeling and hindsight experience replay [30] are used.
In accordance with the interaction metrics during the intrinsic phase (Figs. 4,5), CEE-US and MLP iCEM achieve the highest performance in all four tasks, supporting the benefit of optimizing for long-horizon future novelty. Especially in the object manipulation tasks, data from CEE-US shows clear advantage. Tasks with more objects could not be solved with the amount of data collected.
Table 3: Success rates on tasks with offline RL trained from exploration data generated during the intrinsic phase with the respective method. Using CQL and HER on the data for each task, we see that CEE-US creates much more useful data for offline learning.
Domain
Task
Disagreement [16]
RND [17] ICM [6]
MI.P iCEM
CFF-US
PLAYGROUND
Locomotion
0.05 0.01
0.35 0.05 0.01 0.0
1.0
1.0 0.01
500k datapoints
Relocate obj.
0.06 0.01
0.06
0.06 0.01
0.2 0.06
CONSTRUCTION 600k datapoints
Reach
0.09 + 0.01
0.19 0.05
0.2 + 0.03
0.65+:0.08
0.94 + 0.04
Pick & Place obj.
0.07 = 0.0
0.07 0.0
0.07
0.01
0.18 0.06
0.43 = 0.07
4 Related Work
Intrinsic motivation in RL Prediction error [5, 6, 31]. novelty and Bayesian surprise [20, 32, 33]. learning progress [5, 32, 34], empowerment [35, 36] and count-based metrics [17, 27] are among the popular intrinsic reward signal definitions used in RL. These intrinsic rewards are either used aid
Phttpa://martia 1ab. .o-ggg github. io/ cee-us
We are using the implementation from the d3ripy library [29]
8

---

exploration in challenging tasks with sparse task rewards or in task-agnostic setup where they are the only rewards. Algorithms using the task-agnostic setting follow two routes to solve downstream tasks: (i) they re-lahel the collected data during intrinsic exploration with the downstream task reward and perform offline RL [18, 27, 37] or (ii) they use snapshots of the exploration policy for bootstrapping and fine-tune them on downstream tasks [25, 38]. The first variant does not require additional interaction with the environment, however, additional training of a new policy is still necessary. In addition, offline RL struggles with the distribution shift, while we show that good world model can generalize better. The second variant suffers from another inherent issue: the emerging behaviors of the exploration policy are lost during training. Although Groth et al. [38] address this problem by snapshotting and performing hierarchical RL, this is more of a bandaid solution. Many intrinsically motivated learning systems
goal- -echeiihi setting with predefined goal spaces [32, 34, 39, 40] or auxiliary tasks [41]. The differences in the existing paradigms are highlighted in Table 1. Methods such ICM [6] rely on retrospective intrinsic motivation (here prediction error), i.e. the agent has to already be in the next state to assess how novel that state is. In order to accommodate planning for multi-step intrinsic motivation signals into the future, we need a way to predict them. Haber et al. [42], for instance, do so by learning loss model to predict the prediction error at future time steps. Another line of research, including our method, uses the disagreement of an ensemble of world models to estimate the predictive information gain as the intrinsic motivation measure [16, 18]. Plan2Explore [18] shares similarities to our approach in the task-agnostic exploration phase, using multi-step ensemble disagreement as intrinsic reward. It works with a latent dynamics prediction model and has been applied domains with image observations. However, Plan2Explore does not use structured world models and lacks mechanisms to achieve combinatorial generalization that is beneficial for sample-efficient exploration in object manipulation tasks.
Relational Networks Several works showcase improved dynamics prediction performance in environments with interacting entities using structured world models [43-45]. Kipf et al. [43] uses a GNN to learn the latent transition dynamics in simple manipulation tasks with 2D shapes and 3D blocks from images. However, they use an object-factorized action space and do not tackle exploration but rely on an external dataset. Senncee-GGozzlle et al. [44] applies GNNs to learn the dynamics of physical bodies, where the entities correspond to joints, also without active exploration. Driess et al. [46] uses GNNs and NERFs to obtain object-centric representations from images and RRT planners. Biza et al. [47] also utilizes the combinatorial generalization of structured world models to achieve zero-shot task generalization. However, the policy uses parameterized high-level actions and the world model is learned on an offline expert dataset, thus sidestepping the exploration challenge. Outside the model- paradigm, Li et al. [24] achieves block stacking in CONSTRUCTION using GNN policy, attention and tailored learning curriculum. In contrast, we use GNNs for computing intrinsic rewards and online planning.
Watters et al. [48] deploy curious exploration with retrospective prediction error as an intrinsic reward in an object-centric setting with image inputs and a pre-trained vision model for object discovery, but without object-object interactions. In Seitzer et al. [49], an object-centric causal action-influence is used an intrinsic reward improve sample efficiency.
Curiosity in Object-Centric RL
5 Discussion
In this work, we present CEE-US that combines the learning of GNNs structured world models with curiosity-driven, planning-based exploration. We tackle a significant challenge that is not often addressed in existing intrinsically motivated RL. works: guiding exploration towards potentially useful components of the environment. By approximating information gain via world models injected with relational inductive biases, CEE-US focuses the interactions between entities in the environment. Maximizing future information gain via multistep forward- -aaandin--ninngggg enables CEE-US to discover interaction-rich behaviors more efficiently than the exploration-polin powtecooolllbaaaaeee based basclines, as well as planning-base based paradigms without structured world models.
After the intrinsic free-plat phase, we use the learned GNNs to solve downstream tasks with model-based planning and zero-shot without any additional training. CEE-US can achieve zero- shot generalization without the need for an additional policy learning step, strength of CEE-US that is missing from all the baselines. We show that even non-structured MI.ss achieve zero-shot generalization some tasks, albeit with lower success rates. This indicates that there are benefits

---

to learning good world models and utilizzig them for control even without any further structural biases. This also ties into knowledge-basse intrinsic motivation, as the experience gathered during free play is distilled and stored in the learned world models instead of being dismissed. It is important to note that using model-based planning during the intrinsic phase does not restrict us to model-based approaches in the extrinsic phase. The learned model could also be used to extract task policy later using model-based policy optimization [50] or to perform offline RL on the data generated during free play, which we demonstrate in Sec. 3.4.
Despite the sample efficiency, there are some limitations to model-based online planning. The complexity of behaviors discovered in free play is upper- bounded by the finite planning horizon. The same also applies the extrinsic phase, where solving tasks like throwing objects or solving multistep manipulation tasks require longer planning horizons without extensive reward shaping.
Although CEE-US currently uses proprioceptive state information, do not assume access to any privileged information such that there are no fundamental limitations prohibiting us from applying it to real robots. Following Kipf et al. [43] and Watters et al. [48], CEE-US can also be extended to deal with image inputs and use methods like SCALOR [51] to extract object-centric unsupervised representations. The differentiation between agent and objects can be identified unsupervised [52].
The demonstrated sample efficiency in the unsupervised learning of capable world models opens new avenues for learning directly real hardware. To put the training time into context, 45 training iterations correspond to Ih of interaction time. So our downstream performance in CONSTRUCTION was achieved after about 6.5h of free-play from scratch.
Acknowledgments and Disclosure of Funding
The authors thank Arash Tavakoli and Pavel Kolev for helpful discussions and Andrii Zadaianchuk. Christian Gumbsch and Nico Gürtler for their help reviewing the manuscript. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Cansu Sancaktar and Sebastian Blaes. Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. We acknowledge the financial support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FKZ: 011S (8039B...). This work was supported by the Volkswagen Stiftung (No 98 571).
References
[1] George Loewenstein. The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116(1):55-98, 1994. URL https: //doi org/10 10. 100110000003330 116.116611 1.75.
[2] Cristine H. Legare, Susan A. Gelman, and Henry M. Wellman. Inconsistency With Prior Knowledge Triggers Children's Causal Explanatory Reasoning. Child Development, 81(3): 929 944, 2010. URL https: :/dois org/10. 1111/j 1467-8624 2010. 01443.
[3] Elizabeth Baraff Bonawitz, Tessa J. P. van Schijndel, Daniel Friel, and Laura Schulz. Children balance theories and evidence in exploration, explanation, and learning. Cognitive Psychology, 64(4):215-234, June 2012. URL https: //doi. org/10. /00. 101610////.. cogpsych. 2011 12.002.
[4] Aimee E Stahl and Lisa Feigenson. Observing the unexpected enhances infants' learning and exploration. Science, 348(6230): 94, 2015. URL htttps:///ol org/ 10. 112// science. aaa3799.
[5] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, 1991. URL https: //d1. wtt.aammmm..///////
alc.am.........mmmmmmmmmmmmmcccm org/ org//oii/10/ 5555/
116517. 116542.
[6] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity explo- ration by self-supervised prediction. In International Conference on Machine Learning (ICML), 2017. URL https: http::::////rooednnggem.. mml. mlr. www..ppees///e.mm....m.mmmm prwps/veeee presss/v70/ptthak1772 patp//ataaaaaa... html.
[7] Rachit Dubey and Thomas L. Griffiths. A rational analysis of curiosity. In Proceedings of the 39th Annual Conference of the Cognitive Science Society, pages 307-312, 2017. URL. https:/ / cogsci mindmode ing. org/ 2ttg/201/////
http///apers////// 00t0////000///m/ ieneee//idxx.eeeee html.
10

---

[8] Celeste Kidd and Benjamin Y Hayden. The psychology and neuroscience of curiosity. Neuron,
88(3):449-460 2015. URL https://://do otg/1o. 1t///1..01100 1016/j neuron. 2015. 09. 010.
[9] F Poli, G Serino, RB Mars, and S Hunnius. Infants tailor their attention to maximize learning. Science Advances, 6(39)eabb55033, 2020. URL https: //wp://wwwwwww sciencei org/ htt//doi/ab////i ab.//0.
1126 sttd/vviiiv..... abb5003.
[10] Marshall M Haith. Rules that babies look by: The organization of newborn visual activity.
Lawrence Erlbaum Associates, 1980.
[11] Gregory Murphy and Douglas Medin. The role of theories in conceptual coherence. Psychological review, 92(3):289, 1985. URL attpps:/ddos org/10. 1001 (037/0033-295X 92.
3.289.
[12] Pedro A. Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J. Gershman, and Joshua B. Tenenbaum. Human-level reinforcement learning through theory-based modeling, exploration, and planning. vwXiv:2107.12444 2021. URL https:
/ //arxiv..vv.//////////a//aaaavva org/ aa//abba/2210// 12544.
[13] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Gonzalz, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Re- lational inductive biases, deep learning, and graph networks. https://arze org/ /106. 01261.
2018. URL
[14] Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning (CoRL.), 2020. URL.ttts:: paattss://proceedingg mml. press/ v155/pinns 1/55/pinneri21a.. html.
[15] Marin Vlastelica, Sebastian Blaes, Cristina Pinneri, and Georg Martius. Risk-averse zero- order trajectory optimization. In Conference on Robot Learning (CoRL), 2021. URL https: / /oopnreview
[16] Deepak Pathak, Dhiraj Gandhi. and Abhinav Gupta. Self-supervised exploration via dis- agreement. In International Conference on Machine Learning (ICMI.), 2019. URL https: //proceedings mlr. press / / htt//ptthakke99 html.
[17] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.
URL http::::///peeeevieeee
[18] Ramanan Sckar, Olch Rybkin, Kostas Danilidis, Picter Abbecl, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Confer- ence on Machine Learning (ICML), 2020. URL https::/ mlr. wtt.sprress/v//// sekar20a. html.
[19] E. Pfaffelhuber. Learning and information theory. International Journal of Neuroscience, 3(2):
83-88, 1972. URL https: //doi. orrpli..o..1111/ 1111111..10000010 3118/000074577999111222
[20] Storck, Hochreiter, and Schmidhuber. Reinforcement driven information acquisition in non-deterministic enviromments. In Proceedings of the International Conference on Artificial Neural Newworks, pages 159-164, 164, Paris, 1995. EC2 Cie. URL https: //ppppeee idsia.
ch/"juergen/ cann955ew pdf.
[21] T. M. Cover and J. A. Thomas. Elements of Information Theory, volume 2nd. Wiley, Hoboken, New Jersey, USA, 2006.
[22] D. Y. Little and F. T. Sommer. Learning and exploration in aetion-percepttoon loops. Frontiers
in Neural Circuits, 7(37), 2013. URL https: //doi. .rgo 10. 0389/ffnciz 00037.
[23] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, J. Schneider, Joshua Tobin, Maciek Chociej. P. Welinder, V. Kumar, and W. Zaremba. Multi-gon goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: arXiv:1802.00 094644 2018. URL https: /// /arxivv.... org/ abs.1882..9944.
11

---

[24] Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-objeet manipulation using relational reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4051-4058. IEEE, 2020. URL. htttp:://arxiii
org/abs/ /atbs//99..... 11032.
[25] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark, In NeurIPS 2021 Datasets and Benchmarks Track (Round 2). 2021. URL https: /ttp:////ppeee...eeeem net pet/forummid-lwrrPPe _is.
[26] T. Lillicrap. Jonathan J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. URL https: h //pprr/evee...eeee///// net/ seet/forue ht/f/formmmdd/// ?id=tX_ 180-8Z1.
[27] Nathan Lambert, Markus Wulfmeeer. William F. Whimey, Arunkumar Byravan, Michael Bloesch, Vibhavari Dasagi, Tim Hertweck, and Martin A. Riedmiller. The challenges of exploration for offline reinforeement learning. arXiv:2201.11800 2022. URL https:
//arxiv.org wwtgg//////////eo abs/abb/..../ 11861.
[28] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https:t/////.. http///xxvvvoor//e wwwp.v...r//b////0006 hss//bb//000.....//// 04779.
[29] Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurIPS
2021 Offtine Reinforcement Learning Workshop, 2021. URL httts:://aaxiiii http:/::///xxxoor/aaaa/// orggabs/21111 03788.
[30] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2017. URI. https: //pppe/e////cceggsss neurips pdf.
[31] Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, and Daniel Yamins. Active world model learning with progress curiosity. In International Conference on Machine Learning (ICML), 2020. URL https:://aaaae org/ ha//abb//20000 07853.
[32] Sebastian Blaes, Marin Vlastelica, Jia-Jie Zhu, and Georg Martius. Control What You Can Intrinsically motivated task-planning agent. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL https: http:/:://///ceceeeeeeeeee b6f97e6f0fd1756i 11111111111111771111111
wtwpeeeeiiiinnnnnnnnneeeeeeeeeee www.nnnniiii.c. wetrrisscc/pppeee www.ccccppppp/ee... /209h/as///////
[33] Giuseppe Paolo, Alexandre Coninx, Stephane Doncieux, and Alban Laflaquière. Sparse reward exploration via novelty search and emitters. In Proceedings of the Genetic and Evolutionary
Computation Conference, page 162-162, 2021. URL https://doi. oereii.rrg/10. 10. 1145/3449639. 3459314.
[34] Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre Yves Oudeyer. CURIOUS: Intrinsically motivated modular multi-goal reinforcement learning. In International Conference on Machine Learning (ICML), 2019. URL attpps:// http:///pooe//eeeeeeeeeeee mlr. .press/ v97/colas html.
[35] A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: a universal agent-centric measure of control. In IEEE Congress on Evolutionary Computation, volume 1, pages 128-135 Vol. 2005. URL https: / http//eeeeeeeeeeeeeeee/e ieee. www.eeeeeeeeeeeeeeeeee.eeeeeeeee hedd//uentttte500000
[36] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
//axxiv org/ abs/a//s///// 1000.///0...00 08731.
[37] Denis Yarats, David Brandfonbrener, Hao Liu. Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022. URL. https:////// httttt:s://ppeeeeiiieee
[38] Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Is curiosity all you need? on the utility of
12

---

emergent behaviours from curious exploration. arXiv preprint arXiv:2 2021. URL https://ant http///xxvvv.orr. /ttgg//b////0///// 08603..
[39] OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' Sa, Arthur Petron, Henrique Ponde de Oliveira Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng. Qiming Yuan, Casey Chu, and Wojciech Zaremba. Asymmetric self- play for automatic goal discovery in robotic manipulation. arXiv:2101.04882 2021. URL attps://arze org/ abs//a01.....///////////// 04882.
[40] Russell Mendonca, Olch Rybkin, Kostas Danilidiis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL httpttts://oppereeveew
[41] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference Machine Learning (ICML). 2018. URL htttps:/pproceedingg mlr,
http///////eeemleeeeeeeeeeeeeeee 18a. html.
[42] Nick Haber, Damian Mrowca, Li Fei-Fei, and Daniel K. Yamins. Emergence of structured behaviors from curiosity- -ased intrinsic motivation. In Annual Meeting of the Cognitive Science Society (CogSci), 2018. URL https: /arxivv org/ 1802. 07461.
[43] Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations (ICLR), 2020. URL https: / / //ppnreeeiiie..ccoooo/e wtwpiiiwwwee/ffforumme dd-.ddx66ttda.
[44] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Conference Machine Learning (ICML), 2020. URL https: h/wp:///e/oco.....////o///o mlr, press/
2tass//annhz-gggzaee220aaa gwwggaezzzaea..aaomo html.
[45] Nicholas Watters, Daniel Zoran, Theophane Weber, Peter W. Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from vidco. In Advances in Neural Information Processing Systems (NeurIPS), 2017. URL https://proceediiie www.ediinnnnnuuiss.om www.ine.ucc.oooo httpipps.cc/pape/20////aeeee -wt.ee/t3300cc0eeeebbtbtt.atcttt .html.
[46] Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi- object dynamics with compositional neural radiance fields. arXiv preprint / 2022. URL https: //arxi/v/ org/ abs/ 2202. 11855.
[47] Ondrej Biza, Thomas Kipf, David Klee, Robert Platt, Willem de Meent, and Lawson LS Wong. Factored world models for zero-shot generalization in robotic manipulation. arXiv preprint 220.::2220200333,, 2022. URL https: :/ttppp://rxiv.... org/ abs 05333.
[48] Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and auriosity-driver exploration. arXiv preprint 1xXiv1905.009275 2019. URL https: //arxiv. org/
abs/1905. wtt.:/rg///ab////////// 09275.
[49] Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence detection for improving efficiency in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https: ittpttps h/pp://eedooo.o...ooooo.m www.neeeu.uuuaaappeeeeee wwcpppapeee/2///EL....
c1
pdf.
[50] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2019. URL https: //pppsrs////pp/.///eeee nips.ce, cc/ ptppp/ppee////////////// Aww.raatttttttee..mm.m hhmm..
[51] Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations (ICLR), 2020. URL https: httttps:///peerevieeee wwt.rriwweeeeeeteeeeeeeee tet/ppdffidd-s.Jxrggstttt.
[52] Andrii Zadaianchuk, Georg Martius, and Fanny Yang. Self-supervised reinforcement learning with independently controllable subgoals. In Conference on Robot Learning ( (CoRL), 2021. URI.
https://prooe httpp::/prooeeedings mlr, cress/v164/zadalie htt6:///aaaaaancc/22//aaaaaaaaa http:/aaaaaacccuu2222..2.
pdf.
13

---

[53] Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task reinforcement learning benchmark. htt.c//....ooooooooeeeeeeer/a//a 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] We discuss the limitations of our work in Sec. 5.
(c) Did you discuss any potential negative societal impacts your work? [N/A] We believer there are no negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide the algorithmic details of our method in Sec. 2.4 and further details in supplementary material. The code published on the project website.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Suppl. C
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Further details are in the Suppl. C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Code and videos are published in the project website.
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Wwf4CJ5Mzc9", "reviewer": "Reviewer_zPKR", "summary": "This paper an agent that learns a world model through exploratory play, which it then deploys to solve downstream tasks via a planner. The world model is an ensemble of GNNs, where nodes represent objects, which introduces an object-centric inductive bias. Effective playful exploration is achieved by using disagreement among the ensemble as a proxy for uncertainty, which allows the agent to select actions whose outcomes it cannot yet reliably predict. The authors show that this method works better than baselines, in the sense that it quickly starts engaging in “interesting” interactions (with objects). They do this in two simulated environments with dynamics - a planar world with a variety objects that the agent can push around, and a robot-arm environment where the agent can stack, flip, and throw objects. The authors show that the resulting agent achieves zero-shot generalisation on downstream tasks, to numbers of objects unseen at training time.", "questions": "See below", "limitations": "I believe the ensemble disagreement method that the authors use is susceptible to the white noise problem, wherein the agent is motivated to fixate on dynamics that are random and therefore cannot be learned. Perhaps the authors could comment on this.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper makes progress on an important problem in the RL field, namely learning with intrinsic motivation. As the authors argue, a learned world model that is reward-neutral is a key ingredient for better generalisation and transfer learning. The benefits of learning an object-centric representation in this context are well supported by the authors’ work. The experimental methodology seems sound, and the results are persuasive. The paper is well written and clear and, as far as I can determine, technically sound. Overall I think this is a very good paper that is well above the acceptance threshold for NeurIPS.\n\nI am not sufficiently familiar with the literature on intrinsic motivation to assess the related work section, or to judge the originality of the work with great confidence (hence my “low confidence” rating). However, the following paper got my attention a few years back, which seems very related, and isn’t cited:\n\nNick Haber, Damian Mrowca, Li Fei-Fei, Daniel L. K. Yamins. Emergence of Structured Behaviors from Curiosity-Based Intrinsic Motivation. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018. (https://arxiv.org/abs/1802.07461)\n\nI notice that a subset of the authors of that paper have a more recent ICML paper that also looks relevant, and isn't in the bibliography: https://arxiv.org/abs/2007.07853\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VzM69VGyPKb", "reviewer": "Reviewer_JJXz", "summary": "This paper proposes a curiosity-driven method for exploration via structured world models. It uses GNN ensemble disagreement for computing intrinsic motivation signals in RL, which achieves sample-efficient and interaction-rich exploration in multi-object manipulation environments. Further, the trained world models combined with planning-based methods can achieve zero-shot generalization to many object manipulation tasks. Experiments shows the data-efficiency, the zero-shot generalization ability and manipulation accuracy of the proposed method.\n\n", "questions": "1.\tCould the author show in detail how to train the proposed method using image observations instead of proprioceptive states as the input, and further transfer the proposed method in real-world applications?\n2.\tCould the author show the heaviness (e.g., the training time of the ensemble of M GNNs compared the MLP baseline) of the proposed method? Also, I wonder if it is possible to use some designed intrinsic rewards instead of the ensemble of M GNNs to train a world model with good data-efficiency?\n", "limitations": "The most significant limitation of this paper is the limited application in the real-world setting as the proprioceptive states instead of image observations used in the paper acts as an important constraint.\nBesides, the heaviness in using an ensemble of GNNs may be a limitation. And this limitation may be even serious when the setting becomes more complex, which requires the ensemble to contain more GNNs with more model parameters.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper achieves quite a high sample-efficiency compared with other methods using the curiosity-driven method.\n2.\tThis paper uses GNN with high combinatorial generalization abilities, which makes the method generalizes to environment with novel number of objects.\n3.\tThe world models trained using the proposed method can easily generalize to many downstream manipulation tasks without specific trainings.\n\n\nWeakness:\n1.\tThe world model takes proprioceptive states of objects, instead of the image observation. This may be a hinder to the transfer to the real-world applications, in that: (1) we can not easily get the accurate proprioceptive states of objects; (2) objects with diverse geometries (e.g., real-world cabinets) do not have simple proprioceptive states, and therefore need image observations as input, otherwise the application range will be limited; (3) training using images has a much harder setting than proprioceptive states, which may potentially require the adjustment of the method proposed in this paper. Therefore, in my opinion, it is really important to train the proposed method based on image observations, instead of proprioceptive states.\n2.\tThough resulting in good data-efficiency, using an ensemble of M GNNs makes it heavy to train the model. Training with designed intrinsic rewards (e.g., the number of agent-object / object-object interactions, the state prediction accuracy, etc) may make the training simpler with only one GNN, while keeping the curiosity-driven good data-efficiency.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Uxvdh1KOHan", "reviewer": "Reviewer_Fu6z", "summary": "The authors propose to leverage an ensemble of GNN-based forward-models to estimate an agent's uncertainty with interacting with some part of an object-centric state-space. This uncertainty is used as an intrinsic reward signal during task-agnostic environment exploration with a model (dubbed \"Free Play\"). Given a fully observable MDP, the authors show that this kind of \"Free Play\" enables qualitatively object-centric exploration along with the ability to perform zero-shot generalization to reward functions defined over object attributes.", "questions": "Experiments:\n- Is is possible to run MLP + iCEM but instead of concatenating the object-representations, you simply sum them, or do something that exploits structure but is not a GNN? This would help isolate the benefits of using a GNN.\n\nQuestions:\n- During evaluation, you give the task reward function to a planner in order to select the appropriate action. Your model however, only learns to predict the next state, given an action. It doesn't learn to predict the environment reward---in fact, the agent never sees environment reward. How does the planner know how to leverage the task reward to construct a plan towards that reward? Does the planner have knowledge about all ground-truth states, along with the reward R(s,a) for all ground-truth states?\n- It seems that an MLP leads to the same relative amount of exploration with different objects as the GNN but takes more time to do so? Why do you think it leads to similar results? \n- Figure 3: It seems structured world models lead to object interactions *more quickly*? With infinite time, how do you think of GNN & MLP would have the same asymptotic behavior?\n- What are the numbers in Table 3? Success rate?\n\nBaselines:\n- Plan2Explore seems like a good baseline since it seems similar in principal but doesn't offer combinatorial optimization. Why is this not a baseline? Could you expand on the differences to this baseline?\n\n", "limitations": "Limitations are adequately addressed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Very clear method, results, and introduction\n* High potential for impact in psychology and robotics\n* While the results are somewhat expected, no one has combined disagreement with GNN based methods and the results are convincing\n\n\nWeaknesses:\n* The method has some hand-crafting in parsing out the agent from the objects, though this seems like a fine first-step.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept