Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents a straightforward, but impressive extension of prior work on “Behavior Transformers” for learning policies from offline, reward-free data, by introducing the ability to learn goal-conditioned policies from undirected play data. Crucially, the play data used to learn policies in this work in inherently multimodal (and in some cases suboptimal); yet, using the goal conditioned behavior transformer (C-BeT) approach, we’re able to overcome such problems — *without any online exploration*.

The proposed approach builds on a history of prior work in goal-conditioned behavior cloning from visual observations, and makes a simple tweak to the underlying Behavior Transformer (think a causal transformer that eats states, and outputs actions) architecture, by additionally conditioning on target goal observations (images) of where we’d like the environment to end up. This is a powerful objective, and across three complex simulated environments (CARLA self-driving, multimodal block pushing, and the Franka Relay Kitchen environment) — each with varying levels of multimodality/suboptimality, C-BeT shows itself to be extremely strong.

However, the truly impressive part of this work is the real-robot evaluation; from just 4.5 hours of real user “play” on a Franka robot, C-BeT is able to learn 5 complex manipulation tasks in a “toy kitchen” to a meaningful level of competency — far better than other approaches from the offline RL literature like GO-FAR. In general, I found the evaluation to be thorough, ablations comprehensive, and comparisons to prior art meaningful!
strength_and_weaknesses: I think this is a very strong paper, showing that existing Behavior Transformers can scale to multimodal, reward free play data. In general, I really believe in the results in this paper, and am excited for the possibilities of using such an approach for general offline policy learning on real (and simulated) robotic tasks.

The weaknesses of this work are generally just weaknesses around the feasibility of visual goal-conditioning in the wild. In the real world, it’s not clear whether (given a new task, or a combinatorial explosion of objects/states) we’ll be able to provide robots with meaningful “goal images” that capture what we want to happen; for example, across all the tasks in this work, only a handful of object positions change from start state to goal state; in heavily cluttered environments with dynamic objects, it feels like such an approach would really suffer (in sample efficiency and maybe in just general applicability). This is somewhat shown in the videos, where we add random objects (though that’s also just because the environment is so out of distribution).

clarity,_quality,_novelty_and_reproducibility: I found this paper incredibly clear and easy to read; the quality and thoroughness of the experiments (and attention to baselines) is commendable, and I really buy these results. I like that the authors chose to evaluate on open-source simulated environments as well as a real-robot; it speaks a lot to the reproducibility of this work!

Minor: Seems like Claim 2 in the introduction “C-BeT represents the first work to demonstrate that competent visual policies for real-world tasks can be learned from reward-free play data” isn’t quite right; Implicit BC learns from play-esque multimodal data without rewards, as does follow-up work like Implicit Kinematic Policies (but I could be wrong; might just be a wording thing around the distinction between “play” and “multimodal demos”).

summary_of_the_review: I think this is a straightforward and impactful extension of prior work on behavior transformers. I think that being able to learn robust goal-conditioned policies from uncurated reward-free play data in a manner that is 1) simple and 2) completely offline is very meaningful, and a clear win for the field.

I really like this work!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a goal-conditioned transformer for learning from play data. The approach is a pretty straightforward combination of the Behavior Transformer (BeT) (Shafiullah et al) and goal-conditioning. BeT trains a transformer for imitation learning with a hybrid discrete-continuous action space and has been shown to be effective at capturing multimodal data. Learning from play data is known to be multi-modal, so it makes sense that a goal-conditioned BeT would be an effective way to learn from it. 

The results are pretty strong, outperforming the relevant imitation and offline RL goal-conditioned baselines in a set of 3 simulation environments. Additionally, it has good results on a challenging real robot manipulation task.
strength_and_weaknesses: *Strengths*

- Simple method building on BeT. 
- Intuitive and well-motivated approach for the problem statement (BeT handles multimodality well, makes sense to use it for learning from play data)
- Strong results in simulation and on real hardware.

*Weaknesses*

- Limited novelty beyond BeT, basically just adding goal conditioning. But it's largely an empirical paper and the results are strong so not a big issue.  
clarity,_quality,_novelty_and_reproducibility: Good. 
summary_of_the_review: Overall the paper extends the Behavior Transformer to be goal conditioned. Doing so enables effective goal-conditioned learning from play, outperforming relevant baselines on simulated and real robotics tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method to learn from “play data” in a robot manipulation domain. The “play data” is first popularized by Lynch et al., 2018 and is defined as robot experience data collected through human teleoperation where the human teleoperator controls the robot to achieve a variety of goals in a domain. The difference between such data and, e.g., experience data from reinforcement learning is that neither the underlying goal nor reward signals is given. The proposed method aims to train a policy that (1) captures the multimodal behavior exhibited in such data (2) generate behaviors shown in the training data by conditioning on a desired goal state and (3) generalize to new behaviors not shown in the data. The method combines two prior works: Shafiullah et al. learns multimodal behaviors and is based on a transformer architecture. Lynch et al. (2019) makes the policy goal-conditional. The method is evaluated on a number of simulated domains, including CARLA and a table-top block pushing domain, and a real-world robotics setup where the robot is trained to manipulate objects in a toy kitchen. The paper shows that the proposed method outperforms a few baseline methods, including a prior works without using transformer architecture (Lynch 2019) and a transformer-based model that is not conditioned on the goal (Shafiullah et al., 2022). 
strength_and_weaknesses: Strengths:
+ I like the spirit of developing a unified algorithm for imitation learning from a variety of domains. 
+ I like the real-robot experiment setup and appreciate the effort put into collecting the data and evaluating the policy on a physical robot platform.

Weaknesses:

- My first main concern about the paper is its technical novelty. As stated by the authors themselves, the paper is a somewhat straightforward combination of two prior works, Shafiullah et al., 2021 for the transformer architecture and the multi-modal action learning and Lynch et al., 2019 for goal conditioning. As much as I appreciate the comprehensive empirical study, I had a hard time justifying the technical novelty of this paper given that the main focus of the ICLR conference is on learning methods themselves. 
- My second main concern is the baselines used in the paper. The strongest baselines are BET and Play-GCBC, but they are “set to fail” since the proposed method is the combination of the two methods. Given the baseline choices, I’m not exactly sure the point that the authors hope to convey through the experiment. Is the main contribution of the paper the transformer architecture and the multi-modal behavior policy proposed by BET? Because 5 out of the 6 baselines do not have these components. Is the main contribution of the paper goal-conditioning? Because the strongest baseline BET does not have goal conditioning and obviously will not be able to achieve the specified goal.
- There are also other missing baselines. For example, IRIS [1] and Mandlekar and Xu 2020 [2] are both goal-conditional imitation methods that focus on disentangling multimodal behaviors. Neither is mentioned nor compared. ImplicitBC (Florence et al., 2021) that aims to learn multi-modal behaviors can also be trivially adapted to be conditioned on goals, similar to how the authors adapted BET to be conditioned on goals. 

[1] IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
[2] Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations.


clarity,_quality,_novelty_and_reproducibility: Clarity: good writing. 
Quality: it's a solid execution of a conceptually simple method, although the choice of baselines for empirical evaluation is questionable.
Originality: questionable, as the method is a somewhat straightforward combination of two prior works.
summary_of_the_review: As mentioned in the main comments, although I like the real robot evaluation setup, I have strong doubts about the novelty of the proposed method and the baseline choices in evaluation.

========================
Post-rebuttal review: I thank the reviewer for adding additional baseline results. At the same time, I still believe the research described in this manuscript has limited technical novelty. As much as I'm impressed by the empirical result on real robot experiment, I must consider this factor due to the nature of the venue. I have updated the review score to reflect this.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of offline learning from play data and presents a goal-conditioned version of behavior transformer (BeT) and demonstrates it outperforms several existing algorithms in selected simulation environments and a real robot setup.
strength_and_weaknesses: Strength:
- Goal-conditioned learning is a clean and simple formulation, compared to Q learning.
- C-BeT addresses the shortcomings of BeT by conditioning on goals to resolve ambiguities.
- This paper presents results in both simulated and real-world datasets and shows that C-BeT outperforms baselines in simulated benchmarks.

Weakness: 
- The core idea is doing GCBC with BeT as the BC backbone, which is only incrementally novel. 
- The criteria of choosing baselines is not super clear. A concurrent work of Play-LMP, RIL (Gupta et al.), that outperforms GCBC in the Kitchen domain isn’t included while GCBC is in. A concurrent work of GoFar, PLATO (Belkhale et al.), that outperforms Play-LMP isn’t included while WGCSL (which is shown to be not as good as GoFar) is included.
- The result metric for simulated tasks is confusing, what is the unit? Is it the number of successes over how many total tasks? Why not just use the standard success rate? Did you run multiple different seeds?
- The reason why BeT would outperform goal-conditioned methods isn’t well explained. It is a little counterintuitive and leads to the suspicion that the baseline models are not sufficiently tuned to work with the selected domains. Why not reuse the Play-LMP dataset/tasks? How would C-BeT and BeT perform in tasks without artificially introduced multimodal data?
- The type of generalization is very limited. 
- The setup is fixed and the policy theoretically could have learned the door and knob tasks without the visual observation and be more robust to visual perturbations. 
- The generalization test of “fresh” demonstrations are still of the same tasks.



References:

Gupta, Abhishek, et al. "Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning." Conference on Robot Learning. PMLR, 2020.

Belkhale, Suneel, and Dorsa Sadigh. "PLATO: Predicting Latent Affordances Through Object-Centric Play." arXiv preprint arXiv:2203.05630 (2022).



clarity,_quality,_novelty_and_reproducibility: The paper is well written with clarity and high quality. The code (conditional portion of BeT) is not included in the submission but the hyperparameters are specified in the appendix. 


summary_of_the_review: This paper presents a conditional version of BeT and demonstrates that it performs well on a set of simulated tasks as well as a real robot kitchen environment. The novelty of the method is limited and the selection of baselines is not satisfying but I do appreciate the simplicity of this algorithm. I don’t recommend accepting this work in its current form but won’t argue strongly against it.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work introduces Conditional Behavior Transformers, an approach for conditioning a robot control policy on desired future observations in order to support learning from diverse "play" data.  The resulting conditional policies can be prompted with target observations or demonstration trajectories and, assuming they generalize well, will be able to execute behaviors to achieve the targets.  This approach is demonstrated on simulation and laboratory-based real-world settings.

This paper was met with strongly conflicting initial opinions.  Reviewers ySFW and DhxW found the work straightforward, but impressive due to strong empirical results that worked on a real robot platform.  On the other hand, Reviewers dNx6 and nm8H found the work simple and effective, but too straightforward (i.e., too obvious) and too incremental.  

justification_for_why_not_lower_score: I believe that there is significant value in demonstrating that goal-conditioned learning from play is a powerful approach that can be solved by combining a simple algorithm with the already existing behavior transformer architecture.  I honestly don't see this as incremental given that the learning from play setting is potentially considerably more broadly applicable than the more narrow supervised imitation learning setting.
Decision: Accept: notable-top-5%