From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data
Authors: Zichen Jeff Cui
Yibin Wang
Nur Muhammad Mahi Shafiullah
Lerrel Pinto
Venue: ICLR 2023 notable top 5% (2023)
Keywords: behavior generation
robot manipulation
learning from play

Abstract: While large-scale sequence modelling from offline data has led to impressive performance gains in natural language generation and image generation, directly translating such ideas to robotics has been challenging. One critical reason for this is that uncurated robot demonstration data, i.e. play data, collected from non-expert human demonstrators are often noisy, diverse, and distributionally multi-modal. This makes extracting useful, task-centric behaviors from such data a difficult generative modelling problem. In this work, we present Conditional Behavior Transformers (C-BeT), a method that combines the multi-modal generation ability of Behavior Transformer with future-conditioned goal specification. On a suite of simulated benchmark tasks, we find that C-BeT improves upon prior state-of-the-art work in learning from play data by an average of 45.7%. Further, we demonstrate for the first time that useful task-centric behaviors can be learned on a real-world robot purely from play data without any task labels or reward information. Robot videos are best viewed on our project website: play-to-policy.github.io

Published as a conference paper at ICLR 2023
FROM PLAY TO POLICY: CONDITIONAL BEHAVIOR GENERATION FROM UNCURATED ROBOT DATA
Zichen Jeff Cui
Yibin Wang
Nur Muhammad (Mahi) Shafiullah
Lerrel Pinto
New York University
ABSTRACT
While large-scale sequence modeling from offline data has led to impressive performance gains in natural language and image generation, directly translating such ideas to robotics has been challenging. One critical reason for this is that uncurated robot demonstration data, i.c. play data, collected from non-expert human demonstrators are often noisy, diverse, and distributionally multi-modal. This makes extracting useful, task-centric behaviors from such data a difficult generative modeling problem. In this work, we present Conditional Behavior Transformers (C-Be'T), method that combines the multi-modal generation ability of Behavior Transformer with future-conditioned goal specification. On a suite of simulated benchmark tasks, we find that C-BeT improves upon prior state-of-the-ar work in learning from play data by an average 45.7%. Further, we demonstrate for the first time that useful task-centric behaviors can be learned purely from play data without any task labels or reward information. Robot videos are best viewed our project website: play-to-pol icy. github. ic.
real-world robot
INTRODUCTION
Machine Learning is undergoing Cambrian explosion in large generative models for applications across vision (Ramesh et al., 2022) and language (Brown et al., 2020). A shared property across these models is that they are trained large and uncurated data, often scraped from the internet. Inter- estingly, although these models are trained without explicit task-specific labels in a self-supervised manner, they demonstrate preternatural ability generalize by simply conditioning the model on desirable outputs (e.g. "prompts" in text or image generation). Yet, the success conditional generation from uncurated data has remained elusive for decision making problems, particularly in robotic behavior generation.
To address this gap in behavior generation, several works (Lynch al., 2019; Pertsch al., 2020b) have studied the use of generative models on play data. Here. play data is form of offline, uncurated data that comes from either humans or a set of expert policies interacting with the environment. However, once trained, many of these generative models require significant amounts of additional online training with task-specific rewards (Gupta al., 2019; Singh et al., 2020). In order to obtain task-specific policies without online training, a new line of approaches employ offline RL to learn goal-conditioned policies (Levine et al., 2020; Ma et al., 2022). These methods often require rewards or reward functions to accompany the data, either specified during data collection or inferred through hand-crafter distance metrics, for compatibility with RL training. Unfortunatcly. for many real- world applications, data does not readily come with rewards. This prompts the question: how do we learn conditional models for behavior generation from reward-free, play data?
Table Comparison between existing algorithms to learn from large, uncurated datasets: GCBC (Lynch al., 2019), GCSL (Ghosh et al., 2019), Offline GCRL (Ma et al., 2022), Decision Transformer Chen al. (2021)
GCBC
GCSL
Offline RL
Decision Transformer
C-BeT (ours)
Reward-fres
X /
Offline
X
Multi-modal
X
X
X
*Corresponding author, email: jeff. cui @nyu.edu

---

Published as a conference paper at ICLR 2023
Goal: Turn right knob left knob
Result:
3u00685
trarion
eraeping v. kmob
Suening E. knot
Re-rant
Geosping L. tooe:
Puczing 2. kade
Gcal: Open
uven
Resull:
Success
Re-opaning
Goal: Open
orowave
move pot
Result:
Greaping ricroure
Opaning picrovare
closing aicrovare
Graaping Pet
Transporting Pot
Placing pos
Goal:
Move pot sink
Resuli:
Success
Grasping pot
weaping pot
Goal: Turn left
knob 6 open oven
Reault: Success
Ioteial:zation
Porning left koch
Recracting
Geaeping Oven
Openiing Oran
Ratracting
Goal: Open: oven
Reeult: Failure
Irieialization
Grasping cven
Partially opening
Recracting
Atttepp to zecpen
Riseadd graup
Figure 1: Multiple conditioned roll-outs of visual robot policies Iearned on our toy kitchen with only 4.5 hours of human play interactions. Our model Icams purely from image and proprioception without human labeling data curation. During evaluation, the policy be conditioned either goal observation demonstration Note that last three rows contain distractor objects
environment that were never seen during training.
To answer this question, we turn towards transformer- buusedd generative models that are commonplace in text generation. Here, given prompt, models like GPT-3 (Brown al., 2020) generate text that coherently follow or satisfy the prompt. However, directly applying such models to behavior generation requires overcoming two significant challenges. First, unlike the discrete tokens used in text generation, behavior generation will need models that can output continuous actions while also modeling any multi-modality present in the underlying data. Second, unlike textual prompts that serve conditioning for text generation. behavior generation may not have the condition and the operand be part of the same token set, and may instead require conditioning on future outcomes.
In this work, we present Conditional Behavior Transformers (C-BeT), new model for learning conditional behaviors from offline data. To produce a distribution over continuous actions instead of discrete tokens, C-BeTT augments standard text generation transformers with the action discretization introduced in Behavior Transformers (BeT) (Shafiullah et al., 2022). Conditioning in C-BeT is done by specifying desired future states as input similar to Play-Goal Conditioned Behavior Cloning (Play-GCBC) (Lynch et al., 2019). By combining these two ideas, C-BeT able to leverage the multi-modal generation capabilities of transformer models with the future conditioning capabilities of conditional policy learning. Importantly, C-BeT does not require any online environment interactions during training, the specification of rewards or Q functions needed in offline RI..
2

---

Published as a conference paper at ICLR 2023
We experimentally evaluate C-BeT three simulated benchmarks (visual self-driving in CARLA (Dosovitskiy et al., 2017), molti-modal block pushing (Florence et al., 2021), ) and simulated kitchen (Gupta et al., 2019)), and on real Franka robot trained with play data collected by human volunteers. The main findings from these experiments can be summarized as:
1. On future-conditioned tasks, C-BeT achieves significantly higher performance compared to prior work in learning from play.
2. C-BeT demonstrates that competent visual policies for real-world tasks can be learned from fully offline multi-modal play data (rollouts visualized in Figure 1).
2 BACKGROUND AND PRELIMINARIES
Play-like data: Learning from Demonstrations (Argall et al., 2009) is one of the earliest frameworks explored for behavior learning algorithms from offline data. Typically, the datasets used in these frameworks have a built in assumption that the demonstrations are collected from an expert repeatedly demonstrating single task in exactly the same way. On the contrary, play datasets violate many of such assumptions, like those of expertise of the demonstrator, and the unimodality of the task and the demonstrations. Algorithms that Iearn from such datasets sometimes assume that the demonstrations collected are from a rational agent with possibly some latent intent in their behavior (Lynch et al., 2019). Note that, unlike standard offline-RL. RL datasets (Fu et al., 2020), play-like behavior datasets neither contain fully random behaviors, nor have rewards associated with the demonstrations,
multi-modal behavior cloning model designed particularly for tackling play-like behavior datasets. BeT uses a GPT-like transformer architecture to model the probability distribution of action given a sequence of states π (a; $t-az) from a given dataset. However, unlike previous behavior learning algorithms, BeT does not assume unimodal prior for the action distribution. Instead, it uses a k-means discretization to bin the actions from the demonstration set into & bins, and then uses the bins to decompose each action into discrete and continuous component. This support for multi-modal action distributions make BeT particularly suited for multi-modal, play-like behavior datasets where unimodal behavior cloning algorithms fail. However, vanilla BeTT only supports unconditonal behavior rollouts, which means that it is not possible to choose a targeted mode of behavior during BeT policy execution.
Behavior Transformers (BeT): BeT (Shafiullah et al., 2022)
Conditional behavior learning: Generally, the problem of behavior learning for an agent is con- sidered the task of learning a policy π O - AA mapping from the environment observations the agent's 's actions that elicit some desired behavior. Conditional behavior learning is concerned with learning policy π O x G + conditioned additionally on secondary variable g sampled from distribution p(g). This condition variable could be specific environment states, latents (such as one-hot vectors), or even image observations. The success of a conditioned policy can be evaluated either through pre-specified reward functions, distance function between achieved outcome g' and specified outcome g. or by discounted visitation probability dx(-1g) = Eysz [Ei 7 7'0((((;) = 9)] if a mapping between states and achieved outcome is defined (Eysenbach et al., 2022).
Goal Conditioned Behavior Cloning (GCBC): In GCBC (Lynch et al., 2019; Emmons et al., 2021), I), the agent is presented with dataset of (observation, action, goal) tuples (o, a, g), sequences of such tuples, and the objective of the agent is to learn a goal-conditioned behavior policy. The simplest way to achieve
by training policy T( o, .) that maximizes the probability of the seen data nt" = arg max, , [[(o.a.)) IP(a ~ T(- o, s- Assuming a unimodal Gaussian distribution for T( o. and a model parametrized by 0, this comes down to finding the parameter 0 minimizing the MSE loss, 0* = arg min, Eioa,,) I|a (0,, 0)| To make GCBC compatible with play data that inherently does not have goal labels, goal relabeling from future states is often necessary. A common form of data augmentation in training such models, useful when G C O, is hindsight data relabeling (Andrychowicz et al., 2017), where the dataset {o,, a, g)} is augmented with ((o,, a, or) t f) by relabeling any reached state in future timestep as goal state and adding it to the dataset.
3

---

Published as a conference paper at ICLR 2023
3 APPROACH
Given a dataset ((o, E O x A of sequences of (observation, action) pairs from play dataset, our goal is to learn a behavior generation model that capable of handling multiple tasks and multiple ways of accomplishing each task. At the same time, we wish be able to extract de- sired behavior from the dataset in the form of policy through our model, or, in terms of genera- tive models, "controllably generate" our desired behavior (see Figure 2). Finally, in the process of learning this controllable, conditional gener- ative model, we wish to minimize the amount of additional human annotation or curation re- quired in preparing the dataset. The method we develop address these needs is called Condi- tional Behavior Transformer.
Figure 2: Conditional hehavior learning from play demonstrations. Here, a policy conditioned on reaching ! or @ has only one possible course of action, but con- ditioned on reaching 3 there are two reasonable paths.
3.1 CONDITIONAL BEHAVIOR TRANSFORMERS (C-BET)
Conditional task formulation: First, we formulate the task of learning from a play dataset as learning conditional behavior policy, i.e. given the current state, we need to model the distribution of actions that can lead to particular future states. For simplicity, our formulation can be expressed as O x O + ((A) where, given current observation oe and future observation Og. our policy models the distribution of the possible actions that can take the agent from Og to Oy. Mathematically.
given a set of play trajectories T, we model the distribution T(a Oc, 0g) _ F Frrer(a o. = Te, Og Te, t' t). Next, to make our policy more robust since we operate in the partially observable setting, we replace singular observations with a sequence of observations; namely replacing o, and og with
(1:N) for some integer N. Thus, the final task formulation becomes learning o. = oe (1:N) and 0g- = 0g
generative model with:
Ta | (ll:N) oog.ooo))))) (l-N)) o
& Prer (a (al
(I:N) of = TEiNN. og (I:N) TUN',N, >t)
(I)
Architecture selection: Note that the model for our task described the previous paragraph necessarily multi-modal, since depending on the sequences ō. and og. there could be multiple plausible sequences of actions with non-zero probability mass. As a result, we choose Behavior Transformers (BeT) (Shafiullah al., 2022) our generative architecture base it can learn action generation with multiple modes. We modify the input to the BeT to be a concatenation of our future conditional observation sequence and current observation sequence. We choose to concatenate the inputs instead of stacking them, as this allows us to independently choose sequence lengths for the current and future conditional observations. Since BeT is a sequence-to-sequenee consider the actions associated with the current observations as our actions. We show the detailed architecture of our model in Figure
model, we only
Dataset preparation: To train a C-BeT model on our play dataset {(,, a)}, we will need to appropri- ately prepare the dataset. We first convert the dataset to hold sequences of observations associated with actions, (OE:t- (t:--N) } Then, during training time, we dynamically augment each pair with a se- quence of future observations, functionally converting our dataset into { {Orrr Gest, NN:++N))) for some l' > I, and treat the sequence Osta' + NY og.
Training objective: We employ the same objective as BeT in training C-BeT, For each of the current observation and future conditional pair, compute the BeT loss (see appendix B for details) between the ground truth actions and the predicted actions. We compute the focal loss (Lin et al., 2017) on the predicted action bins, and the MT-loss (Girshick, 2015) on the predicted action offsets corresponding to the action bins as described in BeT.
Test-time conditioning with C-BeT: During test time, we again concatenate future conditional sequence with our current observations, and sample actions from our model according to the BeT framework. While in this work, primarily condition C-BeT future observations, we also study
4

---

Published as a conference paper at ICLR 2023
10g Praiiiin
CAT
Greek Current
"pptl' zuture
Ratirated diateibacion
3chavicr Transforme
Bell Loas (Eocal MZ loaa)
Ground bouth actiona
(Co bee
OF
CNT
Ila R
Target
denonstration
2ase"
AcCLeS distelbutiion
Sarpled actlos
Observations
Condi tioca
Tearsforme
Figure 3: End-to-end training and evaluation of C-BeT. (A) Our dataset consists of play data in environment, which may contain semi-optimal behavior. multi-mode demonstrations, and failures, and not contain any annotations or task labels. (B) We train our C-BeT model by conditioning on current future states using BeT (Section 2) (C) During evaluation, our algorithm can he conditioned by target observations or newly collected demonstrations to generate targeted behavior.
other ways of training and conditioning it, such as binary latent vectors denoting the modes in trajectory in our experiments, and compare its performance to observatioon-coniitooned C-BeT (see Section 4.5).
C-BET ON SIMULATED BENCHMARKS
In this section, we discuss experiments in simulation that designed to answer the following key questions: How well does C-BeT learn behaviors from play? How important is multi-modal action modeling? And finally, how does C-Be'T compare other forms of conditioning?
4.1 BASELINES
We compare with the following sate-offthhean methods in learning from reward-free offline data:
Goal Conditioned BC (GCBC): GCBC (Lynch et al., 2019; Emmons et al., 2021) learns a policy by optimizing the probability of seen actions given current and the end state in trajectory.
Weighted Goal Conditioned Supervised Learning (WGCSL) (Yang et al., 2022): GCSL (Ghosh et al., 2019) is online algorithm with multiple rounds of collecting online data, relabeling, and training policy on that data using GCBC. WGCSL (Yang et al., 2022) improves GCSL by learning additional value function used to weight the GCSL loss. We compare against single-round, offline variant of WGCSL in this work.
Learning Motor Primitives from Play (Play-LMP): Play-LMP (Lynch al., 2019) a behavior generation algorithm that focuses on learning short (~ 30 timesteps) motor primitives from play data. Play-LMP does so by using a variational-autoenoool motor program latents and decoding actions from them.
(VAE) to encode action sequences into
Relay Imitation Learning (RIL): Relay Imitation Learning (Gupta et al., 2019) is a hierarchical imitation learning with high level controller that generates short term target state given long goals, and low level controller that generates action given short term target.
Conditional Implicit Behavioral Cloning (C-IBC): Implicit behavioral cloning (Florence et al., 2021) learns energy based model (EBM) E(a o) over demos and during test samples action given an observation o. We compare against conditional IBC by training an EBM E(a | g).
Generalization Through Imitation (GTI): GTI (Mandlekar al., 2020) encodes the goal condi- tion using a CVAE, and autoregressively rolls out action sequences given observation and goal-latent. We follow their architecture and forgo collecting new trajectories with an intermediate model since that does not fit an offline framework.
Offline Goal-Conditioned RL: While offline RL is generally incompatible with play data without rewards, recently some offline goal-conditioned RI. algorithms achieved success by optimizing for
5

---

Published as a conference paper at ICLR 2023
Table 2: Results of
algorithms on set of simulated environments. The numbers reported for CARLA, BlockPush, and Kitchen are out of 1, I, and respectively, following Shafiullah et al. (2022). In CARLA, success counts as reaching the location corresponding to the observation; for BlockPush, it pushing one or both blocks into the target squares; and for Kitchen, success corresponds to the number of conditioned tasks, out of four, completed successfully.
C-BeT (unimodal)
C-BeT (multimodal)
GCBC
WGCSL
Play-LMP
RIL
C-IBC
GTI
GoFAR
BeT
CARLA
0.04
0.02
0.0
0.59
0.65
0.74
0.72
0.31
0.62
0.98
BlockPush
0.06
0.10
0.02
0.07
0.01
0.04
0.04
0.34
0.35
0.90
Kitchen 0.74
1.17
0.04
0.39
0.13
1.61
1.24
1.77
2.74
2,80
a proxy reward defined through state occupancy. Our baseline, GoFAR (Ma et al., 2022), is one such algorithm that learns goal-conditioned value function and optimizes policy to maximize it.
Behavior Transformers (BeT): We include unconditional BeT (Sec. 2) in our baseline to un- derstand the improvements made by the C-BeT conditioning. In practice, it acts as a "random" " baseline that performs the tasks without regard for the goal.
Unimodal C-BeT: We use method without the multi-modal head introduced in BeT. This also corresponds to a variant of Decision Transformer conditioning on outcomes instead of rewards.
Note that neither WGCSL nor GoFAR are directly compatible with image states and goals, since they require a proxy reward function S x g - IR. Thus, we had to design proxy reward function on the image representations, exp (-//4ll - e1l)2) to apply them on image-hased environments. For fair comparison, we also upgrade baseline Play-LMP, C-IBC, and GTT architectures by giving them sequences of observations and retrofitting them with transformers whenever applicable.
4.2 SIMULATED ENVIRONMENTS AND DATASETS
We run our algorithms and baselines on a collection of simulated environments as a benchmark to select the best algorithms to run on our real robotic setup. The simulated environments are selected to cover a variety of properties that are necessary for the real world environment, such as pixel-based observations, diverse modes in the play dataset, and complex action spaces (see Figure. 4),
1. CARLA self-driving: CARLA (Dosovitskiy et al., 2017) is simulated self-driving enviromment created using Unreal Engine. In this environment, the observations are RGB pixel values of dimension (224, 224, 3). and actions are two-dimensional (accelerate/brake and steer). We an environment with fork in the road (see Figure 2) following two possible routes to the same goal, collecting 200 demonstrations in total. We condition on one of the two possible routes to the goal, and at the goal where choosing either of the two modes is valid.
2. Multi-modal block-pushing We use the multi modal block-pushing environment from Florence al. (2021) for complicated multi-modal demonstrations. In this environment, an xArm robot pushes two blocks, red and green, into two square targets colored red and green. All positions are randomized with some noise at episode start. We use 1,000 demonstrations collected using a deterministic controller, and condition on just the future block positions on each baseline.
3. Franka relay kitchen: Originally introduced in Gupta et al. (2019), Relay Kitchen is robotic environment in a simulated kitchen with seven possible tasks. Franka Panda robot is used to manipulate the kitchen, and the associated dataset comes with 566 demonstrations collected by humans with VR controllers performing four of the seven tasks in some sequence.
4.3 HOW WELL DOES C-BET LEARN BEHAVIORS FROM PLAY?
On each of these environments, we train conditional behavior generation models and evaluate them on a set of conditions sampled from the dataset. The success is defined by the model performing the same tasks as conditioned by the future outcome. We see from Table. 2 that C-BeT performs significantly better compared to the baselines on all three tasks. BeT,, as our unconditioned "random" baseline, shows the success rate of completing tasks unconditionally, and see that none of the baselines surpasses it consistently. Out of the MLP- MLP-based baselines, WGCSL performs best in the baate tasks. However, GoFAR performs best on the CARLA vision based environment where the other two MLP- based baselines fail almost completely. We note that Play-LMP LMP performs poorly because our tasks are long- -oriizon and quite far from its intended motor primitive regime, which may be challenging for Play-LMPPss short-horizon auto-encoding architecture.

---

Published as a conference paper at ICLR 2023
4.4 How IMPORTANT IS MULTI- MODAL ACTION MODELING?
While we use a multi-modal behavior model in this work, it is not immediately obvious that it may be necessary. Specifically, some previous outcome-conditioned policy learning works (Chen et al., 2021; Emmons et al., 2021) implicitly assume that policies are unimodal once conditioned on an outcome. In Table the comparison between C-BeT and unimodal C-BeT shows that this assumption may not be true for all environments, and all else being equal, having an explicitly multi-modal model helps learning an outcome conditioned policy when there may be multiple ways to achieve an outcome.
4.5 How DOES C-BET COMPARE TO OTHER FORMS OF CONDITIONING?
We consider the question of how much comparative advantage there is in getting human labels for our tasks. We do so by adding manual one-hot (CARLA, BlockPush) binary (Kitchen) labels to our tasks, and training and evaluating Table the three C-BeT with those labels. As conditioned on only future simulated environments, C-BeT observations performs comparably to conditioning with human labels.
Table 3: Comparison between C-BeT with supervised labels and labels acquired with human supervision.
No labels
Labels
CARLA BlockPush
0.98
1.0
0.90
0.89
Kitchen
2.80
2.75
5 C-BET ON REAL-WORLD ROBOTIC MANIPULATION
We now discuss our robot experiments, which are geared towards understanding the usefulness of C-BeT on real-world play data.
5.1 ROBOTIC ENVIRONMENT AND DATASET
Robot setup: Our environment consists of Franka Emika Panda robot, similar the simulated Franka Kitchen environment, set up with a children's 's toy kitchen set (see Figure 1). The toy kitchen has an oven, microwave, a pot, and two stove knobs that are relevant to our play dataset. The action joint space in this environment contains the seven joint angle deltas normalized within the [-1, 1] range, and binary gripper control.
We collected 460 sequences totaling to 265 minutes (about 4.5 hours) of play data on the toy kitchen with volunteers using Vive VR controller to move the Franka. While collecting the play data, we did not give the volunteers any explicit instructions about doing any particular tasks, or number of tasks, beyond specifying the interactable items, and stipulating that the pot only goes on the left stove or in the sink, to prevent dropping the pot and reaching an unresettable state. As the observations, we save the RGB observations from two cameras on the left and right of the setup, as well as the robot's proprioceptive joint angles. Overall, the dataset contains 45 287 frames of play interactions and their associated actions.
Play dataset:
Representation learning To simplify the task of learning policies image space, decouple the task of image representation learning from policy learning following Pari et al. (2021). For each camera, first fine-tune a pretrained ResNet-18 (He al., 2016) encoder the acquired frames with BYOL seff-supervision (Grill et al., 2020). Then, during policy learning and evaluation, instead of the image from the cameras, pass the 512-dimensional BYOL embeddings the observation. For the proprioceptive part of the observation, we repeat the (sin, cos) of seven joint states 74 times to get a 1036-dimensional proprioceptive representation, making our overall observation representation 1660-dimensional.
part of
5.2 CONDITIONAL BEHAVIOR GENERATION ON REAL ROBOT
Behavior generation on single tasks: Our first experiment in real robot is about extracting single-task policies from the play dataset. We define our tasks manipulating the four types of interactable objects one at time: opening the oven door, opening the microwave door, moving the pot from the stove the sink, and rotating a knob 90 degrees the right. We use appropriate conditioning frames from our observation dataset. and start the robot from the neutral state to complete the four tasks. The result of this experiment is presented in Table 4. We see that on single task conditionals,
7

---

Published as a conference paper at ICLR 2023
Table Single-task success rate a real world kitchen with conditional models. We present the success rate and number of trials on each task, with cumulative results presented on the last column.
Knobs
Oven
Mierowave
Pot
Cumulative
GoFAR
0/10
0/5
0/5
0/5
0/25
Unconditional BeT
5/20
6/10
1/10
0/10
12/50
Unimodal C-BeT
1/20
8/10
4/10
0/10
13/50
Multimodal C-BeT
3/20
9/10
7/10
5/10
24/50
Table Task success rate in real world kitchen with conditional models evaluated long-horizon goal. present the success rate and number of trials on each task, with cumulative result presented on the last column.
Oven + Pot
Microwave - Oven
Pot + Microwave
Avg. Tasks/Run
Unconditional BeT
(6, 0)/10
6)/0
(0, 1)/10
0.47 0.37
Unimodal C-BeT (1, (1,1)/10
(2, 0)/10
(8, (8,0)/10
Multimodal C-BeT
(5,4)/10
(8, 8)/8)/10
(4, 4)/10
1.1
C-BeT is able to complete all tasks except the knobs consistently, outperforming all our baselines, showing that C-BeT is able to extract single-task policies out of uncurated, real-world play data. We discuss failures of C-BeT on the knob tasks in Section 5.3. While our GoFAR baseline was able to move towards the task targets, it was unable to successfully grasp or interact with any of the target objects. We believe may be the case because unlike the robot experiment in Ma et al. (2022), we do not have the underlying environment state, the tasks are much more complicated, and our dataset is an order of magnitude smaller (400 vs 45 K).
Next, we ask how well our models work for longer- horizon conditioning with multiple tasks. We choose play sequences from the dataset with multiple tasks completed and use their associated states as the conditions for our models. In our roll-outs, we calculate how many tasks completed in the original sequence were also completed in the conditional roll-outs. We calculate this metric over 3 conditioning sequences, and report the results in Table 5. We see that even without any high level controller, C-BeTT is able to stitch together multiple tasks from play demonstrations complete long-horizon goals.
Behavior generation for longer horizons:
Generalization to prompt and environment perturbations: major requirement from any robot system deployed in the real world is to generalize to novel scenarios. We evaluate the generalizability of our learned policies in two different ways. In the first set of experiments, we collect fresh demonstrations that were not in the training set, and we condition our policies on such trajectories. We find that across the different tasks, even with unseen conditionings, C-BeT retains 67% of the single-task performance, with 16/ 50 task successes total. In the second of experiments, we add environmental distractors in the setup (Figure 1, bottom three rows) and run the single- and multi-task conditions on the modified environments. We see once again that the performance drops to around 67% of original with two distractors on the scene, but if we keep adding (four or more) distractors, the robot is unable to complete any tasks.
5.3 ANALYSIS OF FAILURE MODES
We see few failure modes in our experiments that may provide additional insights into learning from real-world play data. We discuss the most salient ones in this section.
Failure in knob operation in the real world: We that in all of our real world experiments, the accuracy in operating the knob is consistently lower than all other tasks. This is due to the failure of the learned representations. Upon inspection of the dataset images' nearest neighbors in the representation space, we see that the BYOL -trinedd representation cannot identify the knob state better than random chance: the returned nearest neighbor differs in knob status often. Since the representation cannot identify the knob status properly, conditioning it naturally fails.
Importance of a multi-modal policy architecture: One of our motivations behind incorporating the BeT architecture in our work is its ability to learn multi-modal action distributions. In our experiments, we show that for some single-task taake conditions such as opening the oven door, having no multi-modall modality is sufficient (Table 4), but for more complicated tasks and learning from more interconnected form of play data, is always the case that a multi-modal architecture prevents our policies from collapsing sub-optimal solutions (Table 5).
8

---

Published as a conference paper at ICLR 2023
6 RELATED WORK
Outcome-conditioned behavior learning: Behavior learning conditioned on particular outcomes, such as reward or goals, is a long studied problem (Kaelbling. 1993; Schaul et al., 2015; Veeriah et al., 2018; Zhao al., 2019). Compared to standard behavior learning, learning conditioned behavior can generally be more demanding since the same model can be expected to learn a multitude of behaviors depending on the outcome, which can make learning long-term behavior harder (Levy et al., 2017; Nachum et al., 2018). As result, a common line of work in nutcome-condition learning is to use some form of relabeling of demonstrations or experience buffer as a form of data augmentation (Kaelbling, 1993; Andrychowicz et al., 2017; Ghosh et al., 2019; Goyal et al., 2022) similar to what we do in the paper. As opposed to goal or state conditioned learning, which we focus on in this paper, recently reward conditioned learning using transformer (Chen et al., 2021) was introduced. However, later work found that it may not work as expected in all environments (Paster al., 2022; Brandfonbrener al., 2022) and large transformer models may not be necessary (Emmons et al., 2021) for reward conditioned learning. In this work, we find that using transformers is crucial, particularly when dealing with high dimensional visual observation and multi-modal actions.
Learning from play data: Our work is most closely related to previous works such as Lynch et al. (2019); Gupta et al. (2019), which also focus on learning from play demonstrations that may not be strictly optimal and uniformly curated for a single task. Learning policies capable of multiple tasks from play data allows knowledge sharing, which is why may be more efficient compared to learning from demonstrations directly (Zhang et al., 2018 8: Rahmatizadeh et al., 201 8; Duan et al., 2017: Pari al., 2021; Young et al., 2021). Gupta et al. (2022) attempts reset-free learning with play data, but requires human annotation and instrumentation in the environment for goal labels.
Generative modeling of behavior: Our method of learning a generative model for behavior learning follows a long line of work, including Inverse Reinforcement Learning IRL (Russell, 1998; Ng et al., 2000; Ho & Ermon, 2016), where given expert demonstrations, a model tries to construct the reward function. which is then used to generate desirable behavior. Another class of algorithms learn generative action decoder (Pertsch et al., 2020a; Singh et al., 2020) from interaction data to make downstream reinforcement learning faster and easier, nominally making multi-modal action distribution easier. Finally, a class of algorithms, most notably Liu et al. (2020); Florence et al. (2021); Kostrikov et al. (2021); Nachum & Yang (2021) do not directly learn generative model, but instead learn energy based models that need to be sampled to generate behavior, although they do not primarily focus on goal-coue condiiiiii.
Transformers for behavior learning: Our work follows earlier notable works in using transformers to learn a behavior model from an offline dataset, such as Chen et al. (2021); Janner et al. (2021); Shafiullah et al. (2022). Our work is most closely related to Shafiullah et al. (2022) as we build on their transformer architecture, while our unimodal baseline is a variant of Chen et al. (2021) that learns outcome conditioned instead of reward conditioned policy. Beyond these, Dasari & Gupta (2020); Mandi et al. (2021) summarizes historical visual context using transformers, and Clever et al. (2021) l) relies on the long-term extrapolation abilities of transformers sequence models. The goal of C-BeT is orthogonal to these use cases, but can be combined with them for future applications.
7 DISCUSSION AND LIMITATIONS
In this work, we have presented C-BeT, a new approach for conditional behavior generation that can learn from offline play data. Across a variety of benchmarks, both simulated and real, we find that C-BeT significantly improves upon prior state-of-the- work. However, we have noticed two limitations in C-BeT,, particularly for real-robot behavior learning. First, if the features provided to C-BeT do not appropriately capture relevant objects in the scene, the robot execution often fails to interact with that object in its environment. Second, some tasks, like opening the oven door, have simpler underlying data that is not multimodal, which renders only meager gains with C-BeTT. A more detailed analysis of these limitations presented in Section 5.3. We believe that future work in visual representation learning can address poor environment features, while the collection of even larger play datasets will provide more realistic offline data for large-scale behavior learning models.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank Sridhar Arunachalam, David Brandfonbrener, Irmak Guzey, Yixin Lin, Jyo Pari, Abitha Thankaraj. and Austin Wang for their valuable feedback and discussions. This work was supported by awards from Honda, Meta, Hyundai, Amazon, and ONR awards N00014-21-1-2558 and N00014- 22-1-2773.
REFERENCES
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay, Advances in neural information processing systems, 30, 2017. 3,9
Brenna D Argall, Sonia Chermova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? arXiv preprint arXiv: wwxiv-2206.01000 2022. 9
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1,
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15014-10097, 2021. 1, 7,9
Henry M Clever, Ankur Handa. Hammad Mazhar, Kevin Parker, Omer Shapira. Qian Wan, Yashraj Narang, Iretiayo Akinola, Maya Cakmak, and Dieter Fox. Assistive tele-op: Leveraging transform- ers collect robotic task demonstrations. arXiv preprint arXiv:2112.05112, 2021. 9
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:2011.05970, 2020. 9
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017. 3,
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning, Advances in neural information processing systems, 30, 2017.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 2021. 3, 7,9
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. arXiv preprint arXiv: Arxiv-2206.07568 2022.
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, arXiv preprint w0X.::2109.001:. 2021. 3, 5,
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals iterated supervised learning. arXiv preprint 1xXiv:9912000888 2019. 5.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015. 4, 13
10

---

Published as a conference paper at ICLR 2023
Anirudh Goyal, Abram Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puig- domenech Badia, Arthur Guez, Mehdi Mirza, Peter C Humphreys, Ksenia Konyushova, et al. Retrieval-augmentt reinforcement learning. In International Conference Machine Learning, pp- 7740-7765. PMLR, 2022. 9
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2122 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:/910. 1956, 2019. 1, 3, 6,
Abhishek Gupta, Corey Lynch, Brandon Kinman, Garrett Peake, Sergey Levine, and Karol Haus- man. Bootstrapped autonomous practicing via multi-task reinforcement learning. arXiv preprint arXiv:2203.15755, 2022. 9
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.7
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, volume 29, pp. 4565-4573, 2016. 9
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 2021. 9
Leslie Pack Kaelbling. Learning to achieve goals. In IN PROC. OF ICAI1993, pp. 1094-1098.
Morgan Kaufmann, 1993. 9
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXtv:2103.080500 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint 0rXiv:2005.01643, 2020.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint 1rXiv:1712.00944 2017. 9
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017. 4, 13
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy- imitation learning. arXiv preprint M0Xxv:2004.09395, 33, 2020. 9
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, S. Levine, and Pierre Sermanet. Learning latent plans from play. Corl, 2019. 1, 9,
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via fiaf-ad f-dvaantage regression. arXiv preprint arXiv:2206.03022 2022. 1, 6,
Zhao Mandi, Fangchen Liu, Kimin Lee, and Pieter Abbeel. Towards more generalizable one-shot visual imitation learning. arXiv preprint arXiv:2 10. 13423, 2021. 9
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Silvio Savarese, and Li Fei-Fei. Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations, arXiv e-prints, art. arXiv:2003.06088, March 2020. 5
Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. arXiv preprini arXiv:2105. /2272, 2021. 9

---

Published as a conference paper at ICLR 2023
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 663-670, 2000. 9
Jyothish Pari, Nur Muhammad, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint 1r 2021. 7,
Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers fail in stochastic environments. arXiv preprint arXiv: Arxiv-2205. 2022.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned
skill priors. arXiv preprint arXiv:2010.11144 2020a. 9
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010./ 2020b.
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based mulk manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3758-3765. IEEE, 2018. 9
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022. 1
Stwart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998. 9
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 07-09 Jul 2015. PMLR. URL https: http://eccceeee...ecee.........e .l www.pnnww................ mml..press/v3.. schaul15 html.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning modes with one stone. arXiv preprint arXiv:2206. /251, 2022. 2, 4, 9, 13, 14
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinchart, and Sergey Levine Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024, 2020. 1,9
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. arXiv preprint arXiv:1806.09605. 2018. 9
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. Iclr, 2022.
Sarah Young, Jyothish Pari, Pieter Abbeel, and Lerrel Pinto. Playful interactions for representation learning. arXiv preprint arXiv: Arxiv-2107 09046, 2021. 9
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5028-56286699 5635. IEEE, 2018.
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum eetropy-re multi-goal reinforcement learning, In International Conference Machine Learning, pp. 7553-7562. PMLR, 2019, 9
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A SIMULATED ENVIRONMENT VISUALIZATIONS
driving
Esanke Kischmn
Figure Visualizations of simulated environments that we evaluate our methods on, from left to right: CARLA self-driving (top down view and agent POV), BlockPush, and Franka Kitchen.
B BEHAVIOR TRANSFORMERS
We use Behavior Transformers from Shafiullah al. (2022) as our backbone architecture, building our conditional algorithm on top of it. In this section, we describe the BeT architecture and the training objective to help the readers understand the details our algorithm.
B. BET ARCHITECTURE
BeT repurposed MinGPT architecture model multi-modal behavior, It uses the MinGPT trunk as a sequence-to-sequees model that tries to predict a sequence of actions amæ4+ given sequence states or observations Oc.thh- Beyond just prediction the actions, however, BeT tries model the multi-moda action distribtuion given the observations. To create multi-modal model over the continuous action distribution, BeT uses an action encoder-decoder architecture that can encode each action vector into a discrete latent and smaller-norm continuous offset. BeT does so by using an offline action dataset to create a k-means model of the actions. Then, an action encoded into its associated bin out of the k-bins (binning), and small continuous offset from the associated bin.
The BeT model, given sequence of observations Oct++h predicts k-dimensional multinomial distribution over the k-bins, as well as & x |A| dimensional matrix for offsets associated with each action bins. Sampling from the BeT action distribution is done via sampling a discrete bin first, taking its associated action offset, and then adding the bin center with the action offset.
B.2 BET TRAINING OBJECTIVE
Given an observation and its associated ground truth action we will now present the simplified version of how the BeT loss is calculated.
Let us assume the BeT model prediction is T(O)a E R&, T()) E | for the discrete and the continuous parts of the predictions. Let us also assume that [a] the discrete bin out of the bins BinCenter((a.)). Then, the Be'T loss becomes
that a belongs to, and (a)
LaST = Lfocat ( (o)a. [a]) + X Lasr (o).)
Where focnt the Focal loss (Lin et al., 2017). a special case of the negative log likelihood loss defined as
Grooal (p:) = -(1 pz)" log(pt)
and I Last the multi-task loss (Girshick, 2015) defined as
MT-Loss a, (aa j=1 Σ
[[[ = 5] (1(a) - (1
13

---

Published as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
C.I IMPLEMENTATION USED
In our work, we base our C-BeT implementation off of the official repo published https: //gittti thub. com/ notmahi /bbet. For the GCBC, WGCSL, and GoFAR baselines, we use the official repo released by the GoFAR authors httpoo/stttts///tttb...oooo http://hhhi.ccmmmmmmmmmamaaaa11
C.2 HYPERPARAMETERS LIST:
We present the C-BeT hyperparameters in Table below, which mostly using the default hyperparameters in the original Shafiullah et al. (2022) paper:
Table 6: Environment-dependen hyperparameters BcT.
Hyperparameter
CARLA
Block-push
Kitchen
Layers
4
6
Attention heads
Embedding width
256
72
120
Dropout probability
0.6
0.1
Context size
10
5
10
Training epochs
40
350
Batch size
128
64
64
Number of bins &
32
24
64
Future conditional frames
10
3
10
The shared hyperparameters are in Table 7.
Table 7: Shared hyperparameters for BeT training
Name
Value
Optimizer
Adam
Learning rate
le-4
Weight decay
0.1
Betas
(0.9, 0.95)
Gradient clip norm
1.0
14

---

Published as a conference paper at ICLR 2023
D ROBOT ENVIRONMENT DEMONSTRATION TRAJECTORIES
Figure 5: Sample demonstration trajectories for the real kitchen environment.
15

---

Published as a conference paper at ICLR 2023
E
SIMULATED ENVIRONMENT ROLLOUT TRAJECTORIES
C-Ber
GC2C
GOFAR
STOR
NGCSL
Figure 6: Sample demonstration trajectories for the CARLA self driving environment, conditioning on going to the right path.
C-BeT
GCBC
GaIAR
WGCSL
Figure 7: Sample demonstration trajectories the modal block pushing environment, conditioning on pushing the green block to green square and red block to red square.
C-Beft
GCBC
GoPAR
WGC31
Figure 8: Sample demonstration trajectories for the Franka Kitchen environment, conditioning on completing the microwave, bottom knob, slide cabinet, hinge cabinet tasks.
16


Reviews:
[{"review_id": "yVGSVX0erh", "reviewer": "Reviewer_nm8H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of offline learning from play data and presents a goal-conditioned version of behavior transformer (BeT) and demonstrates it outperforms several existing algorithms in selected simulation environments and a real robot setup.", "summary_of_the_review": "This paper presents a conditional version of BeT and demonstrates that it performs well on a set of simulated tasks as well as a real robot kitchen environment. The novelty of the method is limited and the selection of baselines is not satisfying but I do appreciate the simplicity of this algorithm. I don’t recommend accepting this work in its current form but won’t argue strongly against it.", "strength_and_weaknesses": "Strength:\n- Goal-conditioned learning is a clean and simple formulation, compared to Q learning.\n- C-BeT addresses the shortcomings of BeT by conditioning on goals to resolve ambiguities.\n- This paper presents results in both simulated and real-world datasets and shows that C-BeT outperforms baselines in simulated benchmarks.\n\nWeakness: \n- The core idea is doing GCBC with BeT as the BC backbone, which is only incrementally novel. \n- The criteria of choosing baselines is not super clear. A concurrent work of Play-LMP, RIL (Gupta et al.), that outperforms GCBC in the Kitchen domain isn’t included while GCBC is in. A concurrent work of GoFar, PLATO (Belkhale et al.), that outperforms Play-LMP isn’t included while WGCSL (which is shown to be not as good as GoFar) is included.\n- The result metric for simulated tasks is confusing, what is the unit? Is it the number of successes over how many total tasks? Why not just use the standard success rate? Did you run multiple different seeds?\n- The reason why BeT would outperform goal-conditioned methods isn’t well explained. It is a little counterintuitive and leads to the suspicion that the baseline models are not sufficiently tuned to work with the selected domains. Why not reuse the Play-LMP dataset/tasks? How would C-BeT and BeT perform in tasks without artificially introduced multimodal data?\n- The type of generalization is very limited. \n- The setup is fixed and the policy theoretically could have learned the door and knob tasks without the visual observation and be more robust to visual perturbations. \n- The generalization test of “fresh” demonstrations are still of the same tasks.\n\n\n\nReferences:\n\nGupta, Abhishek, et al. \"Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning.\" Conference on Robot Learning. PMLR, 2020.\n\nBelkhale, Suneel, and Dorsa Sadigh. \"PLATO: Predicting Latent Affordances Through Object-Centric Play.\" arXiv preprint arXiv:2203.05630 (2022).\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jAPoNzDnOC", "reviewer": "Reviewer_dNx6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to learn from “play data” in a robot manipulation domain. The “play data” is first popularized by Lynch et al., 2018 and is defined as robot experience data collected through human teleoperation where the human teleoperator controls the robot to achieve a variety of goals in a domain. The difference between such data and, e.g., experience data from reinforcement learning is that neither the underlying goal nor reward signals is given. The proposed method aims to train a policy that (1) captures the multimodal behavior exhibited in such data (2) generate behaviors shown in the training data by conditioning on a desired goal state and (3) generalize to new behaviors not shown in the data. The method combines two prior works: Shafiullah et al. learns multimodal behaviors and is based on a transformer architecture. Lynch et al. (2019) makes the policy goal-conditional. The method is evaluated on a number of simulated domains, including CARLA and a table-top block pushing domain, and a real-world robotics setup where the robot is trained to manipulate objects in a toy kitchen. The paper shows that the proposed method outperforms a few baseline methods, including a prior works without using transformer architecture (Lynch 2019) and a transformer-based model that is not conditioned on the goal (Shafiullah et al., 2022). ", "summary_of_the_review": "As mentioned in the main comments, although I like the real robot evaluation setup, I have strong doubts about the novelty of the proposed method and the baseline choices in evaluation.\n\n========================\nPost-rebuttal review: I thank the reviewer for adding additional baseline results. At the same time, I still believe the research described in this manuscript has limited technical novelty. As much as I'm impressed by the empirical result on real robot experiment, I must consider this factor due to the nature of the venue. I have updated the review score to reflect this.", "strength_and_weaknesses": "Strengths:\n+ I like the spirit of developing a unified algorithm for imitation learning from a variety of domains. \n+ I like the real-robot experiment setup and appreciate the effort put into collecting the data and evaluating the policy on a physical robot platform.\n\nWeaknesses:\n\n- My first main concern about the paper is its technical novelty. As stated by the authors themselves, the paper is a somewhat straightforward combination of two prior works, Shafiullah et al., 2021 for the transformer architecture and the multi-modal action learning and Lynch et al., 2019 for goal conditioning. As much as I appreciate the comprehensive empirical study, I had a hard time justifying the technical novelty of this paper given that the main focus of the ICLR conference is on learning methods themselves. \n- My second main concern is the baselines used in the paper. The strongest baselines are BET and Play-GCBC, but they are “set to fail” since the proposed method is the combination of the two methods. Given the baseline choices, I’m not exactly sure the point that the authors hope to convey through the experiment. Is the main contribution of the paper the transformer architecture and the multi-modal behavior policy proposed by BET? Because 5 out of the 6 baselines do not have these components. Is the main contribution of the paper goal-conditioning? Because the strongest baseline BET does not have goal conditioning and obviously will not be able to achieve the specified goal.\n- There are also other missing baselines. For example, IRIS [1] and Mandlekar and Xu 2020 [2] are both goal-conditional imitation methods that focus on disentangling multimodal behaviors. Neither is mentioned nor compared. ImplicitBC (Florence et al., 2021) that aims to learn multi-modal behaviors can also be trivially adapted to be conditioned on goals, similar to how the authors adapted BET to be conditioned on goals. \n\n[1] IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data\n[2] Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CNPjvq5JPX", "reviewer": "Reviewer_ySFW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a straightforward, but impressive extension of prior work on “Behavior Transformers” for learning policies from offline, reward-free data, by introducing the ability to learn goal-conditioned policies from undirected play data. Crucially, the play data used to learn policies in this work in inherently multimodal (and in some cases suboptimal); yet, using the goal conditioned behavior transformer (C-BeT) approach, we’re able to overcome such problems — *without any online exploration*.\n\nThe proposed approach builds on a history of prior work in goal-conditioned behavior cloning from visual observations, and makes a simple tweak to the underlying Behavior Transformer (think a causal transformer that eats states, and outputs actions) architecture, by additionally conditioning on target goal observations (images) of where we’d like the environment to end up. This is a powerful objective, and across three complex simulated environments (CARLA self-driving, multimodal block pushing, and the Franka Relay Kitchen environment) — each with varying levels of multimodality/suboptimality, C-BeT shows itself to be extremely strong.\n\nHowever, the truly impressive part of this work is the real-robot evaluation; from just 4.5 hours of real user “play” on a Franka robot, C-BeT is able to learn 5 complex manipulation tasks in a “toy kitchen” to a meaningful level of competency — far better than other approaches from the offline RL literature like GO-FAR. In general, I found the evaluation to be thorough, ablations comprehensive, and comparisons to prior art meaningful!", "summary_of_the_review": "I think this is a straightforward and impactful extension of prior work on behavior transformers. I think that being able to learn robust goal-conditioned policies from uncurated reward-free play data in a manner that is 1) simple and 2) completely offline is very meaningful, and a clear win for the field.\n\nI really like this work!\n", "strength_and_weaknesses": "I think this is a very strong paper, showing that existing Behavior Transformers can scale to multimodal, reward free play data. In general, I really believe in the results in this paper, and am excited for the possibilities of using such an approach for general offline policy learning on real (and simulated) robotic tasks.\n\nThe weaknesses of this work are generally just weaknesses around the feasibility of visual goal-conditioning in the wild. In the real world, it’s not clear whether (given a new task, or a combinatorial explosion of objects/states) we’ll be able to provide robots with meaningful “goal images” that capture what we want to happen; for example, across all the tasks in this work, only a handful of object positions change from start state to goal state; in heavily cluttered environments with dynamic objects, it feels like such an approach would really suffer (in sample efficiency and maybe in just general applicability). This is somewhat shown in the videos, where we add random objects (though that’s also just because the environment is so out of distribution).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "32ukiA9IDs", "reviewer": "Reviewer_DhxW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a goal-conditioned transformer for learning from play data. The approach is a pretty straightforward combination of the Behavior Transformer (BeT) (Shafiullah et al) and goal-conditioning. BeT trains a transformer for imitation learning with a hybrid discrete-continuous action space and has been shown to be effective at capturing multimodal data. Learning from play data is known to be multi-modal, so it makes sense that a goal-conditioned BeT would be an effective way to learn from it. \n\nThe results are pretty strong, outperforming the relevant imitation and offline RL goal-conditioned baselines in a set of 3 simulation environments. Additionally, it has good results on a challenging real robot manipulation task.", "summary_of_the_review": "Overall the paper extends the Behavior Transformer to be goal conditioned. Doing so enables effective goal-conditioned learning from play, outperforming relevant baselines on simulated and real robotics tasks. ", "strength_and_weaknesses": "*Strengths*\n\n- Simple method building on BeT. \n- Intuitive and well-motivated approach for the problem statement (BeT handles multimodality well, makes sense to use it for learning from play data)\n- Strong results in simulation and on real hardware.\n\n*Weaknesses*\n\n- Limited novelty beyond BeT, basically just adding goal conditioning. But it's largely an empirical paper and the results are strong so not a big issue.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%