PLATO: Predicting Latent Affordances Through Object-Centric Play
Authors: Suneel Belkhale
Dorsa Sadigh
Venue: CoRL 2022 Poster (2022)
Keywords: Human Play Data
Object Affordance Learning
Imitation Learning

Abstract: Constructing a diverse repertoire of manipulation skills in a scalable fashion remains an unsolved challenge in robotics. One way to address this challenge is with unstructured human play, where humans operate freely in an environment to reach unspecified goals. Play is a simple and cheap method for collecting diverse user demonstrations with broad state and goal coverage over an environment. Due to this diverse coverage, existing approaches for learning from play are more robust to online policy deviations from the offline data distribution. However, these methods often struggle to learn under scene variation and on challenging manipulation primitives, due in part to improperly associating complex behaviors to the scene changes they induce. Our insight is that an object-centric view of play data can help link human behaviors and the resulting changes in the environment, and thus improve multi-task policy learning. In this work, we construct a latent space to model object \textit{affordances} -- properties of an object that define its uses -- in the environment, and then learn a policy to achieve the desired affordances. By modeling and predicting the desired affordance across variable horizon tasks, our method, Predicting Latent Affordances Through Object-Centric Play (PLATO), outperforms existing methods on complex manipulation tasks in both 2D and 3D object manipulation simulation and real world environments for diverse types of interactions. Videos can be found on our 
 website: https://sites.google.com/view/plato-corl22/home.

PLATO: Predicting Latent Affordances Through Object-Centric Play
Suneel Belkhale, Dorsa Sadigh Stanford University
Constructing a diverse repertoire of manipulation skills a scalable fashion remains unsolved challenge in robotics. One way to address this chal- lenge is with unstructured human play, where humans operate freely in an en- vironment to reach unspecifted goals. Play is simple and cheap method for collecting diverse user demonstrations with broad state and goal coverage over environment. Due to this diverse coverage, existing approaches for learning from play are more robust to online policy deviations from the offline data dis- tribution. However, these methods often struggle to learn under scene variation and on challenging manipulation primitives, due in part to improperly associating complex behaviors to the scene changes they induce. Our insight is that object- centric view of play data can help link human behaviors and the resulting changes in the environment, and thus improve multi-task policy learning. In this work, we construct latent space to model object affordances properties of an object that define its uses in the environment, and then learn a policy to achieve the desired affordances. By modeling and predicting the desired affordance across variable horizon tasks, our method, Predicting Latent Affordances Through Object- Centri Play (PLATO), outperforms existing methods on complex manipulation tasks in both 2D and 3D object manipulation simulation and real world environments for diverse types of interactions. Videos can be found on our website.
Abstract:
Keywords: Human Play Data, Object Affordance Learning, Imitation Learning
Introduction
The field of robotics has seen tremendous progress in solving manipulation tasks, but learning a gen- eral multi-task policy remains an open challenge. Imitation learning methods are sample-efficient at replicating demonstrated behaviors, but are often presented with structured, predefined tasks and therefore struggle generalize outside of the data distribution [1, 2]. Rather than using predefined task demonstrations, recent work has shown that learning from play data an unstructured form of demonstration without predefined goals can lead to policies that are more robust to online de- viations [3]. Play data is easy to collect at scale since it requires no task specification or manual resetting, and play can have broad data coverage over the set of object interactions necessary for performing variety of tasks.
Existing approaches for learning from play sample short horizon-length windows from play data to learn goal-conditioned imitation policies in an offline fashion [3, 4]. However, not all facets of the demonstrator's behavior captured by the goal alone; thus prior work learns latent space to model the variation in human behaviors. Such a latent space captures a representation of "plans," for example plans representing reaching or grasping motions, for given goal state during play. These latent plans can then help inform robot policies at test time [3].
These approaches make several restrictive assumptions. Firstly, they assume that given fixed short- horizon window of play, the agent's goal is the environment state at the end of the window. This assumption however is not always true: since the desired robot state is not available at test time, the goal is chosen as the environment state a few seconds in the future. Since the goal might be close the same the initial state, it can be uninformative for the policy, does it necessarily represent the human's 's true goal. Crediting behavior to an incorrect environment goal can obscure why human chose the actions during that window, and thus hinder policy learning. For example,
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

if we want to grasp and move a block on a table but our goal is sampled after the robot has initiated motion but before initiating the grasp. the sampled goal will have no change in the object state and thus gives us no information on the true goal that the user had in mind. Secondly, by randomly sampling windows from play data, the learned latent space will be forced to model all sequences of behaviors equally, even though many sequences will be less critical to achieving the desired environment goal. These restrictive assumptions leads to suboptimal behaviors when increasing the complexity and variability of tasks, e.g., tasks with varying horizons, when learning from play.
Instead of defining goals and plans based on arbitrary horizon lengths, we posit that humans often define goals and plan in terms of interactions with objects: rather than planning over the individual joint motions required to grasp and open door handle, we think about turning the handle and then opening the door. Our key insight is that viewing offline play data as diverse object interactions en- ables better modeling of both a human's goals and the behaviors that can achieve these goals. Rather than learning to represent fixed short-horizon robot trajectories, or plans, we bias the latent space towards learning demonstrated object affordances that accomplish tasks in object-space. Imitation policies can then condition these affordances directly to gain insight into human's 's behaviors.
We propose an algorithm, PLATO Predicting Latent Affordances Through Object-Centric Play- that automatically segments play into a series of object interactions using proprioceptive informa- tion, and then learns latent affordance space over object interactions. Simultaneously, learns to imitate human actions over variable horizon interactions, conditioned on the latent affordance and goal. By considering object interactions in play and correctly attributing goals to robot behaviors, PLATO builds robust mapping between the true goals, desired object affordances, and actions on the robot over varying horizons. This leads to PLATO significantly outperforming prior methods especially when increasing the complexity and variability of play data. Our contributions are:
1. We have developed an object-centric paradigm for learning from unstructured human play data, which views play as sequential, unlabeled interactions with objects.
2. We have developed a new algorithm. PLATO, that extracts and leverages these interactions to model diverse object affordances from play data and learn a robust policy.
3. We have tested PLATO on a number of 2D and 3D manipulation environments in simulation and the real world, including diverse objects such as blocks, mugs, cabinets, and drawers, with broad coverage over possible tasks and object affordances. We demonstrate that PLATO substantially outperforms state-of-the oftthe-rrttttttt learning from play baselines in these complex manipulation tasks.
2 Related Work
In this section, we will discuss prior work in goal-conditioned imitation learning. learning from play, and object-centric policy learning. Our work brings an object-centric perspective on learning from play to learn effective imitation policies that generalize across different manipulation tasks.
Imitation Learning. Imitation Learning is a common method for learning robot policies from hu- man demonstrations, where policy learns to mimic human actions [1]. These methods often strug-
gle to generalize to new environments and to learn from multi-task data [5, 6, 2]. To improve gener- alization, Ho, et al. introduced generative adversarial IL to learn from imitation data while matching the expert policy distribution [7]. Recently, implicit imitation learning policies using energy-based models have also been shown to improve generalization as compared to explicit policy models [8]. To enable multi-task learning, one can condition the policy on goal states, either explicitly labelled or inferred via hindsight experience replay [9, 10, 11]. Other methods have leveraged meta-learning for multi-task imitation learning to enable better multi-task performance and one-shot generaliza- tion [12]. A key limitation of these works is the dependency on demonstration quality and quantity, for example the stte-actioon coverage within the dataset for each task [13]. Our work utilizes an imitation learning to learn multi-task policies. However,
approach built on top of goal-conditionas
we use play data opposed to expert demonstrations, which enables a much broader state-action
coverage to learn generalizable and robust policies.
Learning from Play. Human play data is defined as unstructured and unsupervised human interac- tion with an environment, as a means to let the user guide the data collection process, and provide broader coverage of the task-relevant state and goal space. Because humans can freely choose how to interact with the environment, play data is easy and cheap to collect without the need for prior task specification or environment resets. These factors make play data a rich source for skill learning; as
2

---

result, imitation policies trained on play data are more robust to deviations from the expert trajec- tories than those trained on single-task demonstrations [3]. The state of the art method, Play-LMP, learns latent "plans" over short, fixed horizon trajectories from play data [3]. This approach can also be used to ground language with play data using pretrained language models [14]. These learned short-horizon skills have also been chained together to accomplish long horizon tasks using motion planning inspired techniques [15]. Additionally, using small variations in the horizon length of play windows has been shown to improve test time performance [4]. These methods benefit from the broad coverage of play data, but suffer from distribution shift at test time due to incorrectly inferred goals during training. Therefore, our approach also imitates play, but we take an object-centtee view: by learning from diverse interactions with the environment, we can more accurately infer the human's ' goals during play and thus obtain better state-action-gon -oaal distribution.
Object-Centric Modeling and Policy Learning. There is strong evidence that humans have neu- ral pathways specific to object recognition, dynamics understanding, and scene segmentation [16]. Inspired by humans, this notion of object centrism has been applied in many recent works in object manipulation for better policy generalization and sample efficiency. Formalisms like object-oriented MDPs have been introduced to better model effects of agent behaviors as functions of individual ob- jects and their affordances [17]. Object affordances have also been learned from labeled interaction examples in multi-task environments and were shown benefit policy learning downstream [18, 19]. Planning relative to object reference frames and primitives can enable plans to generalize to chang- ing environments [20, 21J. Similarly, focusing on objects during policy learning can allow behaviors to generalize to novel scenarios [22, 23]. A recent work leveraged labelled object motions in play data to predict grasp points, and showed RL becomes more sample efficient post-grasping [24].
We employ object centrism to guide learning from play data, specifically by extracting affordances through unsupervised temporal segmentation over object interactions.
3 Predicting Latent A Affordances Through Object-Centrie Play
A play dataset Dlay consists of N varying length episodes of undirected, human generated state- action trajectories T,. Vj E {I...., .N. where T2 = { S1, for length Ly. The state feature space S may be learned or predefined, and consists of proprioceptive state space S" (robot state space) and an environmental state space S (object state space) such that S @ S = S. The goals that generated these trajectories are not included in the dataset, and are defined as Og E G, for goal space G S 5 We use o and s" interchangeably to refer to environmental state, and likewise for r and s" to refer to the robot state. We assume that goals only consist of the environmental state, since access to proprioceptive state goals at test time is unrealistic as it requires the robot having access to policy for achieving the given environmental goal.
Given access to this play dataset Dpaay, new initial state s E S, and an object goal state Og. our problem is to learn robot policy π to achieve the desired goal state. In prior work, sampled trajec- tories of play T consist of a contiguous fixed-horizon segment from an episode Tj. For simplicity we denote sampling these segments from play as T play. where the length of the segment the fixed horizon H. We propose viewing play sequences from a bird's eye view, namely object interactions, instead of from myopic and fixed 1-2 second windows used in prior work. From this perspective, we hypothesize human play is just a series of environment interactions induced by robot's 's actions. If we can somehow detect where interactions begin and end, we can learn to relate interaction and pre-interaction robot behaviors to the state of the world post-interaction. First, we discuss how we detect interactions in the environment in offline play by leveraging proprioceptive cues. Next, we formalize how we properly choose and associate goals with robot behaviors.
Segmenting Play into Interaction Phases: When we interact with environment, we change its state through our own actions, whether be through direct or indirect contact. In this work, focus single-object interactions, but we emphasize that interactions can be defined even over multiple objects that are being influenced by the robot behavior (e.g., in tool-mediated manipulation, interaction is defined between the tool and the environment). We break down interaction into the following phases:
1. Pre-interaction:
This phase usually involves orienting the robot to interact with an object, e.g., reaching the purple block in Fig. I.
2. Interaction: This phase involves joint and interdependent motions between the robot and the object(s), e.g.. pulling the purple block in Fig. 1.
3

---

Pre-Interaction
Interaction
Dost-Interaction
Ego frei)) reuches book (purple)
Ego puils block
Ego retreats from block
(1) Posterior
(4) Policy on Pre-Interaction
(3) Policy on Interaction
Pre-Interaction Again
Z
f
Present
a
(2) Prior
Z
E'
Z
Condition Policy on Future Affordance
Affordance Provides Context
Our method shown on pre-interaction (purple), interaction (bluc), and post-interaction (green) pe- riods. (1) Posterior E encodes the interaction sequence 7( into affordance z. (2) Prior E' encodes object start and goal states of and Og predict Og is sampled from post- interaction. (3) Policy trained to output actions on interaction period conditioned interaction period conditioned on the "future" affordance. The assumed causal structure shown right: PLATO claims the robot hehavior can be explained with knowledge of the long term goal and future affordance 2: the policy reasons about its desired affordance and goal to determine its actions.
Figure I:
affordance, and simultaneously
output actions from pre- lower
3. Post-interaction: This phase involves separation from the object and any downstream effects on the object, e.g., the purple block coming rest after releasing in Fig. 1. For repetitive interactions, this is the same as the next pre-interaction period.
To account for these periods, we detect the robot's influence over the environment, e.g., grasping a door handle or lifting a mug, to automatically segment play into the periods above. detect- ing single-object interactions, contact signals are readily available with modern robots. The contact signal is smoothed and then chunks of play with contiguous contact are labelled as interaction peri- ods. Several ideas for extending PLATO handle multi-object interaction detection are discussed in detail in Appendix E. Since play consists of many back-to-back interactions, the post interaction phase of one segment, e.g.. retreating from block and moving towards a cup, is simply the pre- interaction for the next segment, e.g., lifting the cup. We denote pre, interaction, and post windows with superscripts (-), (1), and (+), respectively.
Sampling Task Relevant Goals Post-Interaction: By segmenting play into interaction phases, can more accurately sample an accurate goal environment state. Critically, goals result from an interaction by definition. At the end of an interaction, the environment state will very likely have changed. Therefore, rather than arbitrarily labeling the goal as the last state window, in prior work, we can label the goal as any state from the end of the interaction through most-interaction (see 0g in Figure 1). This includes any downstream effects on the object from the interaction (e.g., gravity or inertia). For example, if we slide a block along a table, valid og is any block state before the block stops sliding. With an informative goal sampled, we can proceed to learn goal-conditiones policy.
1-2 second
Extracting Affordances from Interaction for Robust Policy Learning: We define every interac- tion between the robot and the environment as exploiting some affordance on an object in the scene. Affordances are properties of objects that define how they can be used (e.g., a block being grasped, a door knob being turned, or a drawer being opened). Our insight is that learning these affordances (what happens the object) instead of plans (what happens the robot) from play will lead much simpler and more robust task representation that can operate over varying horizons, and thus will yield much better policies at test time. This paradigm empowers the policy to reason about the environment: given access to an affordance (e.g., the door knob being turned) and the goal (e.g, opened door), the policy should be able to work backwards infer the behavior to exploit that fordance (c.g. reach the knob and rotate the gripper to turn it). This in contrast to prior work that

---

relies on randomly selected, short, fixed horizon windows to learn latent representations of plans such plans fail to capture varying horizon tasks and overly depend on the robot state, leading to generalization issues at test time [3].
PLATO Design: The PLATO architecture is shown in Fig. 1. At high level, PLATO learns to model each interaction trajectory 7(9) as latent affordance ((1) Posterior in Fig. 1). This latent affordance & explains the actions both leading up to (pre-interaction) and during the interaction (see causal structure in Fig. 1). For example, knowing that want push block (goal) and how want to push it (affordance) allows the policy to infer that it should servo to the correct side of the block first. Therefore, the latent space is learned end-to-end with the policy æ, which decodes latent affordances conditioned on the current state and object goal states to reproduce both pre-interaction and interaction robot actions ((3) Policy on Interaction and (4) Policy on Pre-Interaction in Fig. 1)- By not learning to encode pre-interaction sequences, the critical assumption we make here is that variations in the affordances (interaction phase) are more relevant the task and thus more important to capture in our latent space than variations across any random behavior sequence (pre-interaction).
The policy action-reconstrn reconstruction objective alone would encourage (plans), leading to a myopic view of the task. In order to force the latent space to focus on object affordances and thus be more robust at test time, we simultaneously learn a prior the affordance distribution conditioned on just the start and goal object states ((2) Prior in Fig. 1), trained to regu- larize the posterior latent affordance z. The posterior on z considers the full window 7(0, while the prior sees just the start and goal object states, and so the prior helps shape the latent space to encode the affordance in 7 rather than just the action information.
model robot behaviors
Algorithm PLATO Training
I: Given: H665, H(-), play data Daay. interaction criteria f(() D( = (Dpay)
2: DS ((0) play play
b. Split into interactions
3: Initialize E, F.,
4: while not converged do
5: ,(-), ,,,,,,,,,,),))))),,,,,,,,,, (+)
((-) D() D(+) play play
6: Sample 0g 1)
7:
p(z) B(r(1))
Posterior Affordance Distribution b Prior Affordance Distribution
8:
p(=')= + E'(o ,0)
9:
2~ p(2)
10:
200
1:R(9 ((s(1)
1, og.
Policy in Interaction b Policy in Pre-Interaction
11:
IR(- + T(8) T (:H1())- 0g, 2)
12:
Compute LPLATO with Eq. (1) and update E, E'.
The training procedure is outlined in Alg. 1. After sampling windows from each interaction phase, r(-), 7(0, and r(+) (Line 5), as well as a long term goal 0g (Line 6), PLATO encodes the interac- tion into a posterior and prior affordance distribution (Lines 7-8). Next an affordance 2 is sampled from the posterior using the reparameterization trick, and then used to decode actions during interaction (Line 10) and pre-interaction (Line 11). See Appendix B. for more discussion of this training procedure, including its computational efficiency. During training, we utilize action recon- struction losses over both the pre-interaction region and the interaction region windows to train all networks end-to-end. We utilize a KL divergence term to both train the affordance prior network and regularize the affordance posterior network.
CPLATO (a Og. 2)) (ag(=(af12)1) og. 2)) + KL(p(2) [I)())) (1)
Here, controls the policy focus on reconstructing pre- pre-interaction behaviors, which usually is set 1. This procedure of learning a posterior and prior network and conditioning the policy on the latent context resembles existing work [3]. The key differences are i) our goals are sampled based on interactions with objects, ii) our latent representations capture object-centric affordances z by intel- ligently shaping the learning objective, which can lead to more effective and generalizable policies, and iii) the policy reasons over varying and longer horizon sequences by conditioning on the same future desired affordance and long term goal 0, throughout both pre- interaction and interaction.
5

---

100
OL LMP ILADOR
PLAIINE
PLAIO
GCBCCED
Fay LMEUO
FLATOUF
80
Success Maie
60-
20-
Puob-1
Push-Rt
Pall-
Pull-t
130
1ip
Side-liot
Poist PushR
Pull-
Tip
Figure 2: Block2D Success Rates, trained over random seeds and evaluated on various primitives for PLATO and baselines Play-LMP and Play-GCBC. Left: Scripted play data, with ablations PLATO-PRE and PLATO-R. Human play data. PLATO substantially outperforms baselines on both scripted and human play data.
Right:
Test Time: At test time, the affordance posterior network E cannot be used to output plans 2, since it assumes access to a trajectory. Similar to prior work [3], the prior network E' is used to propose a' test time, using only the current object state on and the desired goal object state (0₂). Unlike prior work, the prior network only depends on object states, and will be robust to different robot starting states. Therefore instead of having predict the exact robot behavior (i.e., plan) to achieve goal, PLATO predicts an affordance at test time and empowers the policy to exploit this affordance. The policy conditions on z' and goal Og to produce actions at the current state.
Addressing Challenges of Prior Work: By choosing goals og resulting from interactions with ob- jects. PLATO better reflects the true demonstrator goal, and thus can reduce the credit assignment problem found with itly training the policy on re-iinerraction and interaction sequences together, conditioned affordance and long term goal. By biasing the latent space to model affordances during interaction, rather than just any random sequence of play, PLATO learns the task relevant behaviors and varia- tions therein to aid in generalization. Our results in Sec. demonstrate the effectiveness of PLATO compared to prior work wide range of complex tasks. See Appendix A for further discussion.
methods. PLATO handles variable horizon sequences by explic-
the
4 Experiments
In this section, we evaluate our approach extensively across three single-object manipula- tion environments (Block2D, Block3D--latforms, Mug3D-Platforms), one multi-object scene (Playroom3D), and one real scene (Block-Real) across diverse tasks like pushing, lifting, and rotating. These environments, shown in Fig. 3, enable a wide variety of objects and possible affor- dances. We collect scripted play data in all environments as well as human play data for Block2D, and train each method until convergence. See Appendix C for environment, task, data collection, and training details, and Appendix B.2 for method implementation details.
We compare PLATO against the two statc-of-the-ar methods for Learning from Play, Play-GCBC Conditonde BC) and Play-LMP (Latent Motor Plans) [3]. We also implement two variants of our method, PLATO-PRE, which encodes both the interaction and the pre-interaction periods into the latent space (adding r(-) as an input to E in step (1) in Fig. 1), and PLATO-R, which replaces the object-centric prior with the prior from Play-LMP (adding the initial robot state input to E' in Fig. 1). PLATO-PRE results show the sufficiency of affordances as a latent representation, while PLATO-R results show how object-centrism benefits the learned latent space and the policy.
Baselines:
Evaluation results for PLATO for both scripted play data and human play data are shown in Fig. 2, along with results on PLATO-PRE and PLATO-R. On scripted play data, PLATO is able to substantially outperform the basclines on every task. Learning affordances from interactions enables PLATO to model much more complex action sequences, such those involving the tether action, with high accuracy, even across variations in object dimensions, masses, and initial condi- tions. On human play data, we again find that PLATO outperforms Play-GCBC and Play-LMP on all of the tasks, emphasizing the ability of our method to scale to human generated data. Interestingly, performance for all methods is worse the human generated data than scripted data. We attribute this to human data containing many sub-optimal trajectories due to the challenges of teleoperation.
Block2D Results:
PLATO-PRE, which encodes the pre-interaction period, performs slightly worse but similar to PLATO, validating our hypothesis that interaction trajectories (PLATOO) contain sufficient informa- tion about the task when compared to also including pre-interaction trajectories in the latent space

---

Block3D- BlockaD-Platoorm
100
100
50
90-
Saxcess Rate
60-
[10-1
La8-8
Lin-R
LifeF
Foad-L.
Post-R Pusb-F
Poab-L Post-B
Pust-R Push-F
300
MiggD-Plattorms
Sucose Rale
May-GCac
Pay-LMP
PLATO
1054
L10-5
1105-
LIBEF
Rod(1)
Kod(-)
Paab-
Push-B
Pud-R
Push:F
Figure 3: 3D Environment Success Rates, trained over 3 random seeds and evaluated on various primitives for PLATO and baselines Play-LMP and Play-GCBC GCBC, Top Left:
Bottom: Mug3D-Platforms PLATO substantially outperforms baselines in 3D manipulation environments for pushing, rotating, and lifting tasks. Top Right: Block-Real, PLATO trained only in simulation generalizes to real world pushing tasks.
(PLATO-PRE). We find that by adding robot state information the prior (PLATO-R), performance suffers on most tasks. We hypothesize that this phenomenon caused by the prior relying too much the initial state of the robot, and not enough will not generalize at test time when the agent inevitably diverges from the offline state distribution. Overall, we see that framing play data through the lens of object interactions (both PLATO and PLATO-PRE) results in much better policy learning than prior state-of-the-arr methods.
that of the object. As result, the latent space
Fig. 3 shows the results of evaluating PLATO on the harder Platform tasks. Interestingly, Both Play-GCBC and Play-IMP do well on the push- ing tasks in this setting, but do very poorly on the more complex lifting tasks (and rotate tasks for Mug3D). By modeling an affordance space and properly relating these affordances to goal environ- ment states, method is capable of recreating diverse types of varying horizon behaviors from play, and unlike Play-LMP, our method scales smoothly as the number tasks increases.
& Mug3DPlatform Results:
Fig. 4 shows the results on the challenging Playroom3D environment with the drawer, cabinet, and the block, with especially diverse affordances and varying horizon tasks. Once again, PLATO outperforms the baselines on every task. While neither baseline can perform the complex Cabinet-Close primitive, PLATO achieves 100% success. Notably, Play-GCBC performs better than Play-LMP on several tasks, suggesting that the prior network and policy might be out of distribution for the test time states and goals on these complex primitives.
Playroom3D Results:
Block-Real Results: Fig. shows the results of evaluating PLATO on pushing tasks for real robot setup with additional real world data (see Appendix C for details). Play-LMP and PLATO both get near perfect success in simulation since these pushing tasks are less complex, but PLATO generalizes better across the gap between real and simulated object dynamics.
Overall, PLATO achieves substantially higher success rates than the baselines on a wide variety of tasks and object properties in simulation and real environments, along with lower variance across random seeds. This trend is even more apparent when we go beyond simple pushing tasks and consider more complex object interactions such as opening and closing doors, drawers, cabinets, lifting, and others. Due to its object-centric view of play, PLATO is able extract and exploit diverse types of affordances in the environment and can handle varying horizon tasks. See Appendix D for additional experiments including more tasks per scene and longer discussion results.
5 Conclusion
Summary: In this work, we introduced an object-centric paradigm for learning from play data involving segmenting play into series of object interactions. Prior state-of-the-an methods for
7

---

Playroom3D
Play CCBC
100-
Laay
0000
PLATO
Success Rote
60
Open DewwerrOpm
More-ToClbb
Move-To.r
Figure 4: Playroom3D Environment Success Rates, The difference between PLATO and baselines is even greater more complex varying horizon tasks, opening closing drawers cabinets. See Appendix D.I. for additional tasks in this environment involving object retrieval and button pressing.
learning from play suffer from credit assignment issues that stem from the short and fixed horizon of sampled trajectories. Our method, PLATOO, addresses these credit assignment issues by choosing goals that involve meaningful changes in object state. PLATO learns a latent affordance space to model these interactions and their variations, and simultaneously learns to predict these latent affor- dances from the goal object state. These latent affordances help inform the robot behavior across varying horizon tasks. Through our extensive experiments both 2D, 3D, and real world envi- ronments spanning wide variety of object manipulation tasks, we show that PLATO substantially outperforms prior methods on both scripted and human play data.
Limitations and Future Work: object interactions, and our method achieves substantially better performance across a variety of single-object manipulation tasks. In future work, we intend to expand our notion of interaction encompass even nulti-object interactions might have second and third order effects on the environment that could model. We believe the interaction paradigm introduced this work is an important first step to reasoning about these higher order effects. For both the single and multi-object settings, future work might leverage notions of action information density find bottleneck states to automatically segment interactions.
Our work introduces a paradigm of learning from variable horizon
play (e.g.. tool use). When using tools, the robot
As shown in Section 4, data collection methods for play greatly affect final policy performance. We primarily evaluate with scripted play, but we hope collect large human play datasets in future work. Compared to scripted play, we hypothesize that human play consists of much more behavior variability, yielding significant plan and affordance variability for given start and goal state. Iso- lating affordances and interactions helps manage this variability, and PLATO is still able to perform well on all the tasks as result. However, future direction would to study how exactly human play data differs from machine generated play data in order to develop more robust methods.
To add, our real robot experiments demonstrate PLATO trained in simulation can generalize to real world dynamics for pushing tasks without any data from the real robot. We hope to collect play data directly on the robot future work to learn even more complex tasks in the real world. Furthermore, our method makes use of object state information, which may not always be easily available in practice. However, we claim this method can readily handle images with several simple changes to the learned prior, described in Appendix E.
We present thorough discussions of these limitations well as potential solutions in Appendix E.
Acknovledgements
This work was funded by JP Morgan, the Office of Naval Research, and NSF Award Numbers 1849952, 1941722, and 2006388.

---

References
[1] D. A. Pomerlcau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume
1989. URL https: / http//pooeedinngee www.inng...niiisseeeeeeee nwwrips....... httpssccpppppeeppeeeeeeeeeeee 3bw.eeeebcdd3bbbaabbbee
wawpeaerrrrpppppeeo.ae pdf.
[2] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In Y. W. Teh and M. Tit- terington, editors, Proceedings of the Thirteenth International Conference on Artificial In- telligence and Statistics, volume of Proceedings of Machine Learning Research, pages 661-668, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: //proceedings mlr, press /v9 /ttp://///////////// http//oosssoaaaa... html.
[3] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning, pages 1111-1133 PMLR, 2020.
[4] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In L. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30- November I, 2019, Proceedings, volume 100 of Proceedings of Machine Learn- ing Research, pages 1025-1037. PMLR, 2019. URL htt.httpppproceeei.iee httpp/ppoeeenn....ooo www.mngs...........mmm...mmm.... ..r.pres s/v100 gupta20a html.
[5] S. Schaal. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3(6))233-244 242, 1999. ISSN 1364-6613. URL htwps:ttt//ooc www.ssciencee... rect. http:///cciieeeeeearareee///a/// 399013273.
[6] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469 (75))469-483, 2009. ISSN 0921-8890.
httptt/s//dbi.og//0001011.o..o.. com/ http:///ccieeeeerrrrrr//////////
URL https: : /ww. sciencedirect.
[7] Ho and S. Ermon. Generative adversarial imitation learning. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages neurips cc/pappr//01l htwl////hhhhhhho cttp:/cceebbbb/c www.eeccccccccccccc. (w77007774399999 stract.hta.
2016. URL https: https::///c/ceeeeeeeeeee
[8] Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In A. Faust, D. Hsu, and G. Neu- mann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK. volume 164 of Proceedings of Machine Learning Research, pages 158-168 PMLR, 2021. URL https: http:::/pprcceeiingg
mmr. www.mmrrr.ee......mmm.mmmmmmmm.e wwtsss/vv44/fffo/ w01144ffooeecee22. html.
[9] F. Codevilla, M. Müller, A. M. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages 1-9, IEEE, 2018. doi: 10.1109/CCR.200 2.4.4.44...1.8844488. URL https: //tp///ooooooo org.ii.orgg/ 10. 110./119/1//1 IIAO///CRA... 2018 8460 487.
8460487.
[10] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp. Goal-conditioned imitation learning, In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buus Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1 15309, 2019. URL https: http:/poooeeednnggee www.ips.eeuurppcecooe cc/pap e ar/201
c8d3a htt.:///c...............000.a1a.
html.
[11] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, Abbeel, and W. Zaremba. Hindsight experience replay. In Guyon, von Luxburg, S. Bengio. H. M. Wallach, R. Fergus, S. V. Vishwanathan, and R. Garnett, editors, Advances

---

in Neural Information Processing Systems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5048- 5058, 2017. URL httpp:://procceedeetttett htw.:/innnnn.eeueeneeeeeem a3a 3 neurips wwtppppppp......cc.e /201 2ee22//1//hh////// /tt////hhhh///aaaaaaee (aaper//011/aach//555aaaaea
[12] Y. Duan, M. Andrychowicz. B. C. Stadie, J. Ho, Schneider, Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1087-1098, 2017. URL htwphhheebb666000000e500
https: /ttpa::/prroeeee http://pooceennnss///e ngs neurips cC/ 67c1e9575e21 3be0a Abstract. html.
[13] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany. C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Marthn What matters in learning from offline human demonstrations for robot manipulation. Learning, 8-/1 November 1021, London, UK, volume 164 of Proceedings of Machine Learning Research, pages
A. Faust, D. Hsu, and Neumann, editors, Conference on Robot
PMLR, 2021. URL https: s/v166 64 maammmaaa22aaaaammm html.
ngg mlr. pres
[14] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. In D. A. Shell, M. Toussaint, and M. A. Hsieh, editors, Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021. 2021. doi: 10. http://SS.2211XVI111
URL https:
//doi. .r.. 11////1..000 05.001166777 RSS. 2021 XVII. 047.
[15] B. Ichter, Sermanet, and C. Lynch. Broadly-exploring, local-policy trees for long-horizon task planning. In A. Faust, D. Hsu, and G. Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pages 59-69. PMLR, 2021. URL pettps://proceednnng mlr. wwwessppes/////..e..ooo..ee v/l v16414666 ichhe r22a. html.
[16] K. Grill-Spector. The neural basis of object perception. Current Opinion in Neurobiology, 13 (2):159-166, 2003. ISSN 0959-4388. http://sdd...oo//..000000/594530 URL httpt:ttt/... http://wwwecceeiiiececec 803000400.
[17] C. Diuk, A. Cohen, and M. L. Littman. An object-oriented representation for efficient rein- forcement learning. In Proceedings of the 25th international conference Machine learning, pages 240-247, 2008
[18] T. Nagarajan and K. Grauman. Learning affordance landscapes for interaction exploration in 3d environments. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Infor- mation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http://roeednnn....n...
httpppscccpppppppep..ee ppppp/pppp0000 httee//ahhhhhheeeeeeee /ae.1/55222eeee1 5eb335 Abstract
cc13fbb559f1 66ee - html.
[19] A. Khazatsky, A. Nair, D. Jing, and S. Levine. What can I do here? learning skills by imagining visual affordances. In IEEE International Conference on Robotics and Automation, ICRA 2021, Xi' an, China, May 30 June 5, 2021, pages 14291-14297 IEEE, 2021. doi: 20...400000222116699999 URL ht doi org 11w.1...11.IAAAA.. 6. 2021.. 9561692.
N0.1009/CRRA44500.2222
[20] T. Migimatsu and J. Bohg. Object-centric task and motion planning in dynamic environments. IEEE Robotics and Automation Letters, 5(2):844-88) 851, Apr 2020. ISSN 2377-3774. 101:10.1 htt//ra.202.2966550
URL http: /ttp doi /oog///.... 1109 / LRA. 2020 .212222..266 2965875
[21] M. Dalal, D. Pathak, and R. R. Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances Neural Information Processing Systems, 34, 2021.
10

---

[22] C. Devin, P. Abbeel, T. Darrell, and S. Levine. Deep object- representations for gener- alizable robot learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7111-7118, 2018.
8.8611 196.
[23] M. Zhao, Z. Liu, S. Luan, S. Zhang, D. Precup, and Y. Bengio. A consciousnesss-nnpireed plan- ning agent for model-based reinforcement learning. Neural Information Processing Systems, abs/2106.02097, 2021. URL httttts//ppperre nips.
-wstrtttt.bsttttt......... html..
[24] J. Borja-Diaz, O. Mees, G. Kalweit, L. Hermann, J. Boedecker, and W. Burgard. Affordance learning from play for sample-efficient policy learning. In 2022 IEEE International Conference on Robolics and Automation, ICRA. IEEE, 2022. URL U 00352.
[25] Victor Blomqvist. Pymunk 2d physics engine. URL test/index html.
[26] T. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanchez-Gonzalez, E. Grefenstette, P. Kohli, and Battaglia. Compile: Compositional imitation learning and execution. In International Con- ference on Machine Learning, pages 14883428. PMLR, 2019.
[27] R. Ranganath, S. Gerrish, and D. Blei. Black Box Variational Inference. In S. Kaski and Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pages 81 814-822, Reykjavik, Iceland, 22-25 Apr 2014. PMLR. URL htttp://proceeednns ml r. r.press /v33 hatp////annnnaahhhaeeeam 4.hml.
[28] Douglas. Transfer learning from play and language nailing the baseline, Mar 2021. URL https: : / htssshhooooogllaaaao github. 1o/ Learning paarning-fromm from Play//whaaa -oachhatttoo took -us-so-long.


Reviews:
[{"review_id": "yxFKPwqNwhR", "reviewer": "Reviewer_KtMz", "summary": "The authors present a novel algorithm to extract manipulation skills from general purpose play interactions. Play produces a stream of unsegmented interaction data that covers a wide range of behaviors and is easy for a human operator to provide. Previous approaches learn a policy from this data by sampling fixed-length short-horizon windows to learn from. However, this can make learning challenging because the intent of the operator is unknown from this short horizon. The insight presented in this paper is that a large portion of manipulation data can be focused around object interaction. The authors show that segmenting play data based on object interactions, and learning latent-policies based instead off these segmented interactions, leads to improved performance when the policies are evaluated on downstream tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Learning from unstructured play data is a promising way to scale human data collection and the authors do a good job introducing the benefits of this paradigm and the weaknesses of current approaches. Furthermore, the presented insight to naturally segment data from play data using object interactions is intuitive and well presented. The results indeed show that this added structure does help. \n\nThere are two main limitations I would like to discuss followed by some minor comments. First, it is unclear how much performance relies on the quality of interaction segmentation. Is perfect interaction segmentation assumed in this work? How robust is the proposed algorithm to segmentation failure or parameters of the segmentation algorithm? (For example, temporarily broken contact in a push action.) I would guess that the scripted data gives cleaner interaction segmentations while segmentation quality in human collected-data may depend heavily on algorithmic parameters. \n\nThe second limitation I would like to address is that although play data is motivated as an approach to leverage unsegmentated human-collected data, only the Block2D domain is evaluated using human-trajectories. The more challenging domains all use scripted data which makes both action segmentation easier and doesn’t capture additional difficulties that may arise from more variation in human data. That said, I do appreciate that the performance on human data in the Blocks2D domain outperforms the baselines, even if there is a degradation from scripted data.\n\n The following are minor comments that I believe will help improve the clarity of the paper allow the reader to better interpret the results:\n- How is the human-data collected? Who are the users? What instructions were provided to them?\n- I would like to see more discussion about the reliance on the ability to automatically segment play data. How does this limit the proposed approach? Are there tasks that aren’t well modeled this way? What additional infrastructure (software, hardware) is needed for this algorithm to be deployed?\n- Can the authors provide some insight into why the proposed method outperforms the Play-GCBC baseline?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend that this paper be accepted to CoRL. The authors propose an intuitive way to add structure to “learning from play” algorithms which is inspired by the structure common in manipulation tasks. This approach is a promising research direction for more efficiently using human collected data. \n\nPost-Discussion Update:\n\nI would like to thank the authors for taking the time to respond to my review in detail. I appreciate the additional clarity they provided and the many additional experimental results.\n\nThe additional quality of interaction experiments are insightful and do show the algorithm is robust to moderate levels of noise. I would encourage the authors to add motivation for why the fake contacts are only added outside of interactions (in the pre-interaction) as opposed to also breaking contact early during an interaction.\n\nI recommend accepting this paper and encourage the authors to continue investigating the learning from play approaches, especially by including real/unscripted play data for the more complicated tasks.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see above for more details.\n- Dependence on quality of play-data segmentation.\n- Limited results on human collected play data.\n- Minor clarity concerns/requests for more detail.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "sjUYAhVUI6T", "reviewer": "Reviewer_Qcyf", "summary": "This paper proposes a method to learn latent affordance via object-centric play. The key is to segment the play data sequence into with-contact and with-contact. Given this segmentation, it can generate high-quality training data to learn better latent affordance. Several tasks in both simulation and real-world setup are used to evaluate this method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Segmenting the play data is very intuitive, so that better sampling of the initial state and goal state can be achieved for training.\n- Several baselines are compared to demonstrate the performance of the proposed method.\n\nWeaknesses:\n- All experiments use only one object. It will be much better to evaluate the method on a complex environment with multiple objects, for example, the play data used in play-LMP.\n- Segment play data based on contact is intuitive for short-horizon motion requiring only one contact. But if a complex primitive requires contact switching, for example, switching between different tools, this paradigm is hard to handle this kind of long-horizon motions.\n- Some paragraphs need move revision to improve the clarity. For example, the paragraph \"Learning Affordances during Interaction\" in Section 3 is hard to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The overall intuition to segment play data based on contact is good. But the experiments' setup is a bit toy. It will be better to demonstrate the proposed method in a complex environment. Some revisions are required to improve the presentation clarity. The proposed method is hard to be applied to complex primitives requiring multiple contacts.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "It will be better to demonstrate the proposed method in a complex environment.\nSome revisions are required to improve the presentation clarity.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "igkYVV-bXW", "reviewer": "Reviewer_Z8qY", "summary": "This paper proposes a framework for learning goal-conditioned policies from play data. The key insight is to segment the play data based on the robot's contact with objects, instead of splitting the data into fixed-horizon episodes. As a result, each segment of the play data now has a meaningful goal based on the change of object states (referred as \"affordance\" in the paper), which can be encoded into a latent vector upon which a policy can be conditioned.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Very nice presentation. Motivation is clear throughout the paper.\n- Simple but effective algorithm design. The meaning of latent codes is much clearer compared to prior work like Play-LMP.\n\nWeaknesses\n- Only works on a single-object scene.\n- Requires ground-truth object state estimation.\n- Not much technical novelty compared to prior work. The contribution is mainly on segmenting the data properly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although I like the idea of using object states to segment play data, I don't think the contribution of this paper is enough for a good conference paper.\n\nI always think the fixed horizon episodes in play-LMP don't make sense. This paper addresses this issue nicely and demonstrates the benefits of using object-conditioned latent representation. However, it is a pity that the method only works on single-object environments and requires ground-truth object states (e.g. 6D poses), which is very hard to acquire in real environments with clutter. Even the simulation environments themselves are quite simple. The variety of objects is very poor (only blocks and mugs). And it is not clear how different is the test environments from the training ones. The method still has a long way to go from being useful in real-world manipulation.\n\nIn addition, I have doubts about the benefits of using play data. It is supposed to contain more diversity and easier to collect. However, in many of the papers that use play data, the play is scripted (meaning diversity is limited) and the environment is fixed. Why not scaling the data collection to more diverse environments and diverse objects? With these simple environments, I don't see the point of having human \"play\" with it. For most of these simple tasks like pushing and lifting, collecting expert demonstrations per task can be equally or more efficient.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- How are the object states $o$ represented in the network? It should be made clear whether they are explicit states like 6D poses or more like raw RGB(D) observations. These make two very different problems.\n- The method segments play into episodes based on contact, and claims that contact detection is easy in modern robots (line 158). However, in multi-object interaction scenarios (such as using tools mentioned in line 146), the environment (object states) can change without the robot directly contacting another object. How to detect the change in goals in these situations?\n- Why does it make sense to use latent code from the future for the pre-interaction policy?\n- The paper shows that the prior latent code $z'$ is important, but how important is the posterior code $z$? What if the policy network directly conditions on the prior code $z'$?\n- In Fig. 2 and 3 there are tasks like Push-L, Push-B, etc. The meaning of \"-L\" etc. is not explained. I assume it means left, back, right and forward. Why are these considered different tasks and evaluated separately? The performance also varies a lot (80% for Lift-L and 100% for Lift-B in Fig. 3), but they seem the same task to me."}}, {"review_id": "_d6dETTcI3yB", "reviewer": "Reviewer_7Bgr", "summary": "- Learn object affordances through unstructured human play, then train a policy to achieve the affordances.\n- Key insight: viewing offline play data as diverse object interactions yields better modeling of goals and behaviors. Using automatic segmentation of play into a series of object interactions using proprioceptive information.\n- Model: like in [3], learn prior and posterior networks that predict an affordance latent z using KL-divergence. Posterior sees sequence of interaction, while prior only sees start and goal states. The action decoding policy is only trained on the prior latent to ensure that it remains \n- Advantages over previous method [3]:\n  - dynamic horizon as opposed to fixed short term horizon\n  - less credit assignment issues\n  - prior more robust to robot starting states\n- Envs: block 2d, block 3d, mug 3d, playroom 3d, block real, tasks: pushing, lifting, rotating\n- Results: play used as object interactions results in much better policy learning than baselines.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n- strong results (e.g. much higher accuracy than baselines in playroom) and convincing evaluation\nit does well across a variety of tasks, playroom is diverse and hard\n- probably the most scalable strategy for diverse data collection method for the real world that exists today, so even though the setups are narrow in the paper, this is a very important problem that is very relevant beyond these narrow setups\n- nice implementation details in appendix.\n- impressive playroom videos (sim)\n\nweaknesses\n- real-world setup not very impressive\n- not using image inputs\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This line of work has potential for major impact because play data might currently be the most scalable strategy for acquiring diverse robotics skills. This paper has some limitations (no vision and limited real setup), but addresses some fundamental problems in earlier learning from play methods and thus brings the field closer to being able to scalably acquire massive sets of diverse skills, which will be a turning point for the field. The model is sound, well explained and yields strong results and robust policies.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- “Contact signals are readily available with modern robots. The contact signal is smoothed and then chunks of play with contiguous contact are labelled as interaction periods.”\n  - Can you clarify exactly what signal is used (in both real and sim)? torque sensing? pressure sensing (if so specify the limitation that the back of the end effector may not have pressure sensing and cannot be used during play to push things around for example)?\n  - if this info is in appendix and I missed it, maybe put a reference to it in the main body?\n- “Unlike prior work, the prior network only depends on object states, and will be robust to different robot starting states. Therefore instead of having to predict the exact robot behavior (i.e., plan) to achieve a goal, PLATO predicts an affordance at test time and empowers the policy to exploit this affordance.”\n  - is this true from pixels? if not, how do you plan on addressing that when switching to vision? This is an important problem to address.\n  - or do you mean that z is more object-focused?\n- performance is worse with human data, due to challenges of teleoperation: does that challenge the entire play approach? or having better teleop should fix it?\n"}}]

Author rebuttal:

Decision: Accept (poster)