Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a framework for learning goal-conditioned policies from play data. The key insight is to segment the play data based on the robot's contact with objects, instead of splitting the data into fixed-horizon episodes. As a result, each segment of the play data now has a meaningful goal based on the change of object states (referred as "affordance" in the paper), which can be encoded into a latent vector upon which a policy can be conditioned.
strengths_and_weaknesses: Strengths
- Very nice presentation. Motivation is clear throughout the paper.
- Simple but effective algorithm design. The meaning of latent codes is much clearer compared to prior work like Play-LMP.

Weaknesses
- Only works on a single-object scene.
- Requires ground-truth object state estimation.
- Not much technical novelty compared to prior work. The contribution is mainly on segmenting the data properly.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Although I like the idea of using object states to segment play data, I don't think the contribution of this paper is enough for a good conference paper.

I always think the fixed horizon episodes in play-LMP don't make sense. This paper addresses this issue nicely and demonstrates the benefits of using object-conditioned latent representation. However, it is a pity that the method only works on single-object environments and requires ground-truth object states (e.g. 6D poses), which is very hard to acquire in real environments with clutter. Even the simulation environments themselves are quite simple. The variety of objects is very poor (only blocks and mugs). And it is not clear how different is the test environments from the training ones. The method still has a long way to go from being useful in real-world manipulation.

In addition, I have doubts about the benefits of using play data. It is supposed to contain more diversity and easier to collect. However, in many of the papers that use play data, the play is scripted (meaning diversity is limited) and the environment is fixed. Why not scaling the data collection to more diverse environments and diverse objects? With these simple environments, I don't see the point of having human "play" with it. For most of these simple tasks like pushing and lifting, collecting expert demonstrations per task can be equally or more efficient.
issues: - How are the object states $o$ represented in the network? It should be made clear whether they are explicit states like 6D poses or more like raw RGB(D) observations. These make two very different problems.
- The method segments play into episodes based on contact, and claims that contact detection is easy in modern robots (line 158). However, in multi-object interaction scenarios (such as using tools mentioned in line 146), the environment (object states) can change without the robot directly contacting another object. How to detect the change in goals in these situations?
- Why does it make sense to use latent code from the future for the pre-interaction policy?
- The paper shows that the prior latent code $z'$ is important, but how important is the posterior code $z$? What if the policy network directly conditions on the prior code $z'$?
- In Fig. 2 and 3 there are tasks like Push-L, Push-B, etc. The meaning of "-L" etc. is not explained. I assume it means left, back, right and forward. Why are these considered different tasks and evaluated separately? The performance also varies a lot (80% for Lift-L and 100% for Lift-B in Fig. 3), but they seem the same task to me.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a novel algorithm to extract manipulation skills from general purpose play interactions. Play produces a stream of unsegmented interaction data that covers a wide range of behaviors and is easy for a human operator to provide. Previous approaches learn a policy from this data by sampling fixed-length short-horizon windows to learn from. However, this can make learning challenging because the intent of the operator is unknown from this short horizon. The insight presented in this paper is that a large portion of manipulation data can be focused around object interaction. The authors show that segmenting play data based on object interactions, and learning latent-policies based instead off these segmented interactions, leads to improved performance when the policies are evaluated on downstream tasks.
strengths_and_weaknesses: Learning from unstructured play data is a promising way to scale human data collection and the authors do a good job introducing the benefits of this paradigm and the weaknesses of current approaches. Furthermore, the presented insight to naturally segment data from play data using object interactions is intuitive and well presented. The results indeed show that this added structure does help. 

There are two main limitations I would like to discuss followed by some minor comments. First, it is unclear how much performance relies on the quality of interaction segmentation. Is perfect interaction segmentation assumed in this work? How robust is the proposed algorithm to segmentation failure or parameters of the segmentation algorithm? (For example, temporarily broken contact in a push action.) I would guess that the scripted data gives cleaner interaction segmentations while segmentation quality in human collected-data may depend heavily on algorithmic parameters. 

The second limitation I would like to address is that although play data is motivated as an approach to leverage unsegmentated human-collected data, only the Block2D domain is evaluated using human-trajectories. The more challenging domains all use scripted data which makes both action segmentation easier and doesn’t capture additional difficulties that may arise from more variation in human data. That said, I do appreciate that the performance on human data in the Blocks2D domain outperforms the baselines, even if there is a degradation from scripted data.

 The following are minor comments that I believe will help improve the clarity of the paper allow the reader to better interpret the results:
- How is the human-data collected? Who are the users? What instructions were provided to them?
- I would like to see more discussion about the reliance on the ability to automatically segment play data. How does this limit the proposed approach? Are there tasks that aren’t well modeled this way? What additional infrastructure (software, hardware) is needed for this algorithm to be deployed?
- Can the authors provide some insight into why the proposed method outperforms the Play-GCBC baseline?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend that this paper be accepted to CoRL. The authors propose an intuitive way to add structure to “learning from play” algorithms which is inspired by the structure common in manipulation tasks. This approach is a promising research direction for more efficiently using human collected data. 

Post-Discussion Update:

I would like to thank the authors for taking the time to respond to my review in detail. I appreciate the additional clarity they provided and the many additional experimental results.

The additional quality of interaction experiments are insightful and do show the algorithm is robust to moderate levels of noise. I would encourage the authors to add motivation for why the fake contacts are only added outside of interactions (in the pre-interaction) as opposed to also breaking contact early during an interaction.

I recommend accepting this paper and encourage the authors to continue investigating the learning from play approaches, especially by including real/unscripted play data for the more complicated tasks.

issues: Please see above for more details.
- Dependence on quality of play-data segmentation.
- Limited results on human collected play data.
- Minor clarity concerns/requests for more detail.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: - Learn object affordances through unstructured human play, then train a policy to achieve the affordances.
- Key insight: viewing offline play data as diverse object interactions yields better modeling of goals and behaviors. Using automatic segmentation of play into a series of object interactions using proprioceptive information.
- Model: like in [3], learn prior and posterior networks that predict an affordance latent z using KL-divergence. Posterior sees sequence of interaction, while prior only sees start and goal states. The action decoding policy is only trained on the prior latent to ensure that it remains 
- Advantages over previous method [3]:
  - dynamic horizon as opposed to fixed short term horizon
  - less credit assignment issues
  - prior more robust to robot starting states
- Envs: block 2d, block 3d, mug 3d, playroom 3d, block real, tasks: pushing, lifting, rotating
- Results: play used as object interactions results in much better policy learning than baselines.

strengths_and_weaknesses: strengths
- strong results (e.g. much higher accuracy than baselines in playroom) and convincing evaluation
it does well across a variety of tasks, playroom is diverse and hard
- probably the most scalable strategy for diverse data collection method for the real world that exists today, so even though the setups are narrow in the paper, this is a very important problem that is very relevant beyond these narrow setups
- nice implementation details in appendix.
- impressive playroom videos (sim)

weaknesses
- real-world setup not very impressive
- not using image inputs

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This line of work has potential for major impact because play data might currently be the most scalable strategy for acquiring diverse robotics skills. This paper has some limitations (no vision and limited real setup), but addresses some fundamental problems in earlier learning from play methods and thus brings the field closer to being able to scalably acquire massive sets of diverse skills, which will be a turning point for the field. The model is sound, well explained and yields strong results and robust policies.
issues: - “Contact signals are readily available with modern robots. The contact signal is smoothed and then chunks of play with contiguous contact are labelled as interaction periods.”
  - Can you clarify exactly what signal is used (in both real and sim)? torque sensing? pressure sensing (if so specify the limitation that the back of the end effector may not have pressure sensing and cannot be used during play to push things around for example)?
  - if this info is in appendix and I missed it, maybe put a reference to it in the main body?
- “Unlike prior work, the prior network only depends on object states, and will be robust to different robot starting states. Therefore instead of having to predict the exact robot behavior (i.e., plan) to achieve a goal, PLATO predicts an affordance at test time and empowers the policy to exploit this affordance.”
  - is this true from pixels? if not, how do you plan on addressing that when switching to vision? This is an important problem to address.
  - or do you mean that z is more object-focused?
- performance is worse with human data, due to challenges of teleoperation: does that challenge the entire play approach? or having better teleop should fix it?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method to learn latent affordance via object-centric play. The key is to segment the play data sequence into with-contact and with-contact. Given this segmentation, it can generate high-quality training data to learn better latent affordance. Several tasks in both simulation and real-world setup are used to evaluate this method.
strengths_and_weaknesses: Strengths:
- Segmenting the play data is very intuitive, so that better sampling of the initial state and goal state can be achieved for training.
- Several baselines are compared to demonstrate the performance of the proposed method.

Weaknesses:
- All experiments use only one object. It will be much better to evaluate the method on a complex environment with multiple objects, for example, the play data used in play-LMP.
- Segment play data based on contact is intuitive for short-horizon motion requiring only one contact. But if a complex primitive requires contact switching, for example, switching between different tools, this paradigm is hard to handle this kind of long-horizon motions.
- Some paragraphs need move revision to improve the clarity. For example, the paragraph "Learning Affordances during Interaction" in Section 3 is hard to follow.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The overall intuition to segment play data based on contact is good. But the experiments' setup is a bit toy. It will be better to demonstrate the proposed method in a complex environment. Some revisions are required to improve the presentation clarity. The proposed method is hard to be applied to complex primitives requiring multiple contacts.
issues: It will be better to demonstrate the proposed method in a complex environment.
Some revisions are required to improve the presentation clarity.
review_title: null

Meta-review: metareview: The paper provides an interesting concept of using object-centric play data for learning multi-task policies with a sound framework. The paper is well presented and the reviewer concerns were well addressed including improving the experiments section. Including more complex tasks remains a concern and it can significantly improve the paper, if addressed. However, with the current experiments, the paper makes sufficient contributions to the community.     
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)