Behavior Transformers: Cloning $k$ modes with one stone
Authors: Nur Muhammad Mahi Shafiullah
Zichen Jeff Cui
Ariuntuya Altanzaya
Lerrel Pinto
Venue: NeurIPS 2022 Accept (2022)
Keywords: Behavioral cloning
learning from demonstrations

Abstract: While behavior learning has made impressive progress in recent times, it lags behind computer vision and natural language processing due to its inability to leverage large, human-generated datasets. Human behavior has a wide variance, multiple modes, and human demonstrations naturally do not come with reward labels. These properties limit the applicability of current methods in Offline RL and Behavioral Cloning to learn from large, pre-collected datasets. In this work, we present Behavior Transformer (BeT), a new technique to model unlabeled demonstration data with multiple modes. BeT retrofits standard transformer architectures with action discretization coupled with a multi-task action correction inspired by offset prediction in object detection. This allows us to leverage the multi-modal modeling ability of modern transformers to predict multi-modal continuous actions. We experimentally evaluate BeT on a variety of robotic manipulation and self-driving behavior datasets. We show that BeT significantly improves over prior state-of-the-art work on solving demonstrated tasks while capturing the major modes present in the pre-collected datasets. Finally, through an extensive ablation study, we further analyze the importance of every crucial component in BeT. Videos of behavior generated by BeT are available here: https://mahis.life/bet

Behavior Transformers: Cloning k modes with one stone
Nur Muhammad (Mahi) Shafiullah
Zichen Jeff Cui
Ariuntuya Altanzaya
Lerrel Pinto
New York University
Abstract
While behavior learning has made impressive progress in recent times, it lags behind computer vision and natural language processing due to its inability to leverage large, human-generated datasets. Human behaviors have wide variance, multiple modes, and human demonstrations typically do not come with reward labels. These properties limit the applicability of current methods in Offline RL and Behavioral Cloning to learn from large, pre-collected datasets. In this work, we present Behavior Transformer (BeT), new technique model unlabeled demon- stration data with multiple modes. BeT retrofits standard transformer architectures with action discretization coupled with a multi-task action correction inspired by offset prediction in object detection. This allows us to leverage the multi-mode modeling ability of modern transformers to predict multi-modal continuous ac- tions. We experimentally evaluate BeT on a variety of robotic manipulation and self-driving behavior datasets. We show that BeT significantly improves over prior state-of-the-ar work on solving demonstrated tasks while capturing the major modes present the pre-collected datasets. Finally, through an extensive ablation study, analyze the importance of every crucial component in BeT. Videos of behavior generated by BeT are available here: https:/ //mahis...././///e life/bet.
Introduction
Creating agents that behave intelligently in complex environments has been longstanding problem in machine learning. Although Reinforcement Learning (RL.) has made significant advances in behavior learning, its success comes the cost of high sample complexity [57, 23, 1]. Without priors on how to behave, state-of-the-art RI. methods require online interactions on the order of 1-10M 'reward-labeled" samples for benchmark control tasks [81]. This is in stark contrast vision and language tasks, where pretrained models and data-driven priors are the norm [19, 11, 32, 6]. which allows for efficient downstream task solving.
So how do we learn behavioral priors from pre-collected data? One option is offline RL [47], where offline datasets coupled with conservative policy optimization can learn task-specific behaviors. However, such methods have yet to tackle domains where task-specific reward labels are not present. Without explicit reward labels, imitation learning. particularly behavior cloning, is more fitting option [68, 9, 77]. Here, given hehavior data D = {st, at), behavior models can be trained to predict actions fo(st) a; through supervised learning. When demonstration data is plentiful, such approaches have found impressive success a variety of domains from self-driving [68, 14] to robotic manipulation [84, 62]. Importantly, requires neither online interactions nor reward labels.
"Coresponding author, email: mahi@cs. nyu.edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

CARLA Rollout
Start
Turn left
Turn right
Turn right
Turn right
Trajectory
Rollout 2
Start
Turn right
Turn left
Turn Left
Turn left
Trajectory
Rollout
Block Push
Et
Start
Reach red
Push red to red
Reach green
Push green to green
Rollout
Start
Reach green
Push green to red
Reach red
Push red to green
Ro lout 1
Franka Kitchen
Start
Microwave
Kettle
Bottom knob
Hinge
Rollout 2
Start
Kettle
Light swi tch
Slide
Hrigee
Figure I: Unconditional rollouts from BeT models trained from multi- demonstartions the CARLA, Block push, and Franka Kitchen euvironments. Due the multi-modal architecture BeT, in the same environment successive rollouts can achieve different goals or the same goals in diffcrent ways.
However, statc-of-the-att behavior cloning methods often make a fundamental assumption that the data drawn from a unimodal expert solving single task. This assumption is often baked in the architecture design, such as using Gaussian prior. On the other hand, natural pre-collected data is sub-optimal, noisy, and contains multiple modes of behavior, all entangled in a single dataset. This distributionally multi-modal experience is most prominent in human demonstrations. Not only do we perform a large variety of behaviors every day, our personal biases result significant multi-modality even for the same behavior [31, 52]. Current approach for behavior cloning from such datasets primarily focus on learning goal-conditioned policies, where each goal implies a single mode of behavior [35, 34, 52, 16). However, even after goal-conditioning, an important question remains: How do we train models that can natively "clone" multi modall behavior data?
In this work, we present Behavior Transformers (BeT),, new method for learning behaviors from rich, distributionally multi-modal data. BeT is based of three key insights. First, we leverage the context based multi-token prediction ability of transformer based sequence models [78] to predict multi- modal actions. Second, since transformer based sequence models are naturally suited to predicting discrete classes, we cluster continuous actions into discrete bins using k-means [53]. This allows us model high-dimensional, continuous multi-modal action distributions as categorical distributions without learning complicated generative models [42, 20). Third, to ensure that the actions sampled from BeT are useful for online rollouts, we concurrently learn residual action corrector to produce continuous actions for sampled action bin.

---

We experimentally evaluate BeT on five datasets ranging from simple diagnostic toy datasets to complex datasets that include simulated robotic pushing [25], sequential task solving kitchen environments [34], and self-driving with visual observations in CARLA [21]. The two main findings from these experiments can be summarized as:
1. On multi-modal datasets, BeT achieves significantly higher performance during online rollouts compared to prior behavior modelling methods.
2. Rather than collapsing or latching onto one mode, BeT able cover the major modes present in the training behavior datasets. Unconditional rollouts from this model can be seen in Fig. 1.
All of our datasets, code, and trained models will be made publicly available.
Behavior Transformers
Given a dataset of continuous observation and action pairs D behaviors we are interested in, our goal is to learn a behavior policy π + A that models this reward labels. This setup follows
{(o, a)) C O x A that contains
data without any online interactions with the environment the Behavior Cloning formulation, where policies are trained to model demonstrations from expert rollouts. Often, such policies are chosen from hypothesis class parametrized by parameter set Following this convention, our objective is to find the parameter 0 that maximizes the probability of observed data
0- : arg max
P(aa| 02:0)
(1)
8
When the model class is restricted to unimodal isotropic Gaussians, this maximum likelihood estimation problem leads to minimizing the Mean Squared Error (MSE), E. || 0)12.
Limitations of traditional MSE- based BC: While MSE-based BC has been able to solve variety of tasks [9, 77], it assumes that the data distribution is unimodal. Clean data from an expert demonstrator solv- ing particular task in particular way satisfies this assumption, but pre- collected intelligent behavior often may not [52, 34]. While more re- cent behavior generation models have sought to address this problem, they often require complex generative mod- els [76], an exponential number of bins for actions [54], complicated training schemes [66], or time-consuming test-time optimization [25]. An experimental analysis of some of these prior works is presented in Section 3.
Unimodal
Figure Comparison between regular MSE-bar ASseb BC model and BeT models can capture multi-modal distributions, The MSE-BC model takes action to minimize MSF.
Overview of Behavior Transformers (BeT): We address two critical assumptions in regular BC. First, we relax the assumption that the behavior we are cloning is purely Markovian, and instead model P(a, 0N, 0t-1. 0t-A+1) for some horizon h. Second, instead of assuming that actions are generated by a unimodal action distribution, we model our action distribution as a mixture of gaussians. However, unlike previous efforts similar Mixture Density Networks (MDN) to do so, whose limitations have been explored in Florence et al. [25], we do not explicitly predict mode centers, which significantly improves our modeling capacity. To operationalize these two features in single behavior model, we make use of transformers since (a) they are effective in utilizing prior observational history, and (b) they are naturally suited to output multi-modal tokens through their architecture.
2.1 Action discretization for distribution learning
Although transformers have become standard as backhone sequence-to-seqqeeee models [19, 11], they designed to process discrete tokens and not continuous values. In fact, modeling multi-modal

---

Continuous action binning datatet ((9) 9
Transforer training
Par-clasa actton
K Wino
Offict aead
Grousd truch class
VinGPT
8.4
0.0 0.5
deservatton Sequence
8in probo
Ground truth
categerical actionn bin
M.ky
Contfrunss actioo
Ground truth actton a.
action (1 6]
Ground trutt actiion
andder
cacoder
C. Test tine rollouts
Actton offaste
sumpled
WinaFT
cecodar
Observation fon
Bin probebriittie
Sampted bin
Figure 3: Architecture of Behavior Transformer. (A) The continuous action binning using k-means algorithm that lets BeT split every action into discrete bin and a continuous offset, and later combine them into one full action. (B) Training BeT using demonstrations offline; each ground truth action provides ground truth bin and residual action, which used train minGPT trunk with its binning and action offset heads. (C) Rollouts from BeT in test time, where it first chooses a bin and then picks the corresponding offset to reconstruct a continuous action.
distributions nigh-dimensional continuous variables tractable manner is in itself a challenging problem, especially if we want the trained behavior model to cover the modes present in the dataset. To address this, we propose new factoring of the action prediction task by dividing each action in two parts: a categorical variable denoting an 'action center', and a corresponding 'residual action".
To this end, given the actions in dataset, first optimize for set of i action centers, {41, A2, ... , C A. We then decompose each action into two parts: a categorical variable representing the closest action bin, [a] := arg min, ||a A:ll2, and a continuous residual action Ala. If we are given the set of action centers TA,ke, the residual action (a), we can deterministically reconstruct the true action learned, these k-means based encoder and decoders for this action factorization process are fixed for the rest of the train and testing phases. The action factorization procedure is illustrated in Fig. 3 (A).
(a)
action bin index [a] and
ALoJ (a). Once
2.2 Attention-based behavior mode learning
Once we have the clustering based autoencoder learned from the actions in the dataset, we model our demonstration trajectories with BeT. We use transformer decoder model, namely minGPT [11], with minor modifications, as our backbone. The transformer takes in a sequence of continuous obser- vations (oi, 0i+1, 0;- 1) and learns sequence-to-sequee
model mapping each observation to categorical distribution over discrete action bins. The predicted probability sequence is then compared with the ground truth labels, ([a,], [4++1] [a.+2]. (04+h-1]). We use negative log-likelihood based Focal loss [49] between the predicted categorical distribution probabilities and the ground truth labels train the transformer head. Focal loss is a simple modification over the standard cross entropy loss. While the standard cross entropy loss for binary classification can be thought of Lee (pt) log(pt), Focal loss adds term (1 p.)" to this, make the loss
L = 7 log(pt)
This loss has the interesting property that its gradient is more steep for smaller values pt, while flatter for larger values of pt. Thus, it penalizes and changes the model more for making emrors in the low-probabiliity classes, while is more lenient about making errors in the high probability classes. The model is illustrated in Fig. 3 (B).

---

2.3 Action correction: from coarse to finer-grained predictions
Using a transformer allows us to model multi-modal actions. However, discretizing the continuous action space in any way invariably causes loss of fidelity [39]. Discretization error may cause online rollouts the behavior policy to go out of distribution from the original dataset [73], which can in turn cause critical failures. To predict the complete continuous action, add extra head to the transformer decoder that offsets the discretized action centers based on the observations.
For each observation o; in the sequence, the head produces l x (im(A) matrix with & proposed k
(a )) where the residual actions residual action vectors, (fal j-1 ((a, ())) (at (,())) (a;
correspond to bin centers A1, A2, A3, Ag. These residual actions are trained with a loss akin to the masked multitask loss [30] from object detection. In case, the ground truth action is a, the loss is:
MT-Loss a, ((a (1()
Σ T[[a] = -) (l(a) - (a j=1
Where [I[] denotes the Iverson bracket, ensuring the offset head of BeT only incurs loss from the ground truth class of action a. This mechanism prevents the model from trying to fit the ground truth action using the offset at every index.
2.4 Test-time sampling from BeT
During test time, at timestep t we input the latest h observations (or, 11-1.-.. ,0t- to the transformer, combining the present observation o, with h 1 previous observations. Our trained MinGPT model gives / & bin center probability vectors, and h dim( offset matrix. To sample an action at timestep we first sample an action center according to the predicted bin center probabilities on the gl index. Once we have chosen an action center Ar,j, we add the corresponding
residual action (a to it to recover predicted continuous action á; Ae.j (a This sampling procedure is illustrated in Fig. 3 (C).
3 Experiments
We now study the empirical performance of BeT on variety of behavior learning tasks. Our experiments are designed answer the following questions: (a) Is BeT able to imitate multi-modal demonstrations? (b) How well does BeT capture the modes present in behavior data? (c) How important are the individual components of BeT?
3.1 Environments and datasets
We experiment with five broad environments. While full descriptions of these environments, dataset creation procedure, and overall statistics are in Appendix A, brief description of them are as follows.
(a) Point mass environment #1: Our first set of experiments in Fig. 2, used to get a qualitative understanding of BeТ, were performed a simple Pointmass environment with a 2D observation and action space with two hundred demonstrations. The pre-collected demonstrations start at fixed point, and then make their way to another point while avoiding block in the middle. The two primary modes in this dataset are taking left turn versus right turn.
(b) Point mass environment #2: The setup is similar to the previous environment with the exception of one straight line and two complicated prolonged 'Z' shaped modes of demonstration (Fig. 5.)
(c) CARLA self-driving environment: CARLA [21] uses the Unreal Engine to provide sim- ulated driving environment in visually realistic landscape. The agent action space is 2D (accelerate/brake and left/right steer), while the observation space is 244.224.)-dimmesioona RGB image from the car. hundred total demonstrations drive around a building block in two distinct modes. This environment highlights the challenge of behavior learning from high-
dimensional observations as shown in Fig. (a). For visual observations with BeT, we use frozen ResNet-18 18 [36] pretrained on ImageNet [18] an encoder.
5

---

(d) Multi-modia block-pushing environment: For more complicated interaction data, we use the multi- molli-modal block-pushing environment from Implicit Behavioral Cloning (IBC) [25], where an XArm robot needs to push two blocks into two squares in any order. The blocks and target squares are colored red and green. The positions of the blocks are randomized at episode start. We collect 1,000 demonstrations using deterministic controller with two independent axes of multi-modality: (a) starts by reaching for either the red or the green block, with 50% probability, and (b) it pushes the blocks to (red, green) or (green, red) squares respectively with 50% probability.
(e) Franka kitchen environment: To highlight the complexity of performing long sequences of actions, we use the Relay Kitchen Environment [34] where Franka robot manipulates virtual kitchen environment. We use the relay policy learning dataset with 566 demonstrations collected by human participants wearing VR headsets. The participants completed sequence of four object-interaction tasks in each episode [34]. There are a total of seven interactable objects in the kitchen: microwave, kettle, slide cahinet, a hinge cabinet, light switch, and two burner knobs. This dataset contains two different kinds of multi-modality: one from the inherent noise in human demonstrations, and another from the demonstrators' intent.
3.2 Baseline behavior learning methods
While full description of our baselines are in Appendix B.I, brief description of them is here:
(a) Multi-layer Perceptron with MSE (RBC): We use MLP networks trained with MSE loss as our first baseline, since this is the standard way of performing behavioral cloning for a new task [77]. A comparison with rransfommr-bbsee behavior cloning is discussed in Section 3.5.
(b) Nearest neighbor (NN): Nearest neighbor based algorithms are easy implement, and has recently shown to have strong performance on complicated behavioral cloning tasks [3].
(c) Locally Weighted Regression (LWR): This non-parametric approach provides better regular- ization compared to NN and is a strong alternative parametric BC [4, 62].
(d) Variational auto-ensoders (VAE): Inspired by SPiRL [66], where behavioral priors are learned through a VAE [42], we compare with continuous actions generated from the VAE and the prior.
(e) Normalizing Flow (Flow): Inspired by PARROT [76]. where state-conditioned action priors are learned through Flow model [20], we compare with actions generated from the Flow model.
(f) Implicit Behavioral Cloning (IBC): Instead of modeling the conditional distribution P(a o). IBC models the joint probability distribution P(a, o) using energy based models [25]. While IBC is slower than explicit BC models because of their sampling requirements, they have been shown to learn well on multi-modal data, and outperform earlier work such as MLP-MDN [8].
3.3 Is BeT able imitate multi-modal demonstrations?
The first question we ask is whether BeT can actually clone behaviors given mixed dataset of unlabeled, multi-modal behaviors. To examine that, we look at the performance of our model in CARLA, Block push, and Kitchen environments compared with our baselines in Table 1.
We see that BeТ outperforms all other methods in all environments except CARLA, where it is narrowly outperformed by LWR. Since the models are all behavioral cloning algorithms, they share the failure mode of failing once the observations go of distribution (OOD). However, they vary in the tolerance. For example, BeT shines in the Block push environment, where alongside extreme environment randomness and multi-modality, the models also have to learn significant long-term behaviors and commit to a single mode over a long period. While all basclines can somewhat successfully reach one block, they fail to complete the long-horizon, multi-modal task of pushing two blocks into two different bins. On the other hand, we observe that BeT's primary failure mode is not realizing a block has not completely entered the target yet, while other methods either go OOD quickly, or keep switching between modes. We also observe that BeT performs well even in complex observation and action spaces. In the CARLA environment, the model takes in visual observations, while in the Franka Kitchen environment, the action space corresponds to 9-DOF torque controlled robot. BeT handles both cases with the same ease as does environments with lower-dimensional observation action spaces.

---

Table 1: Performance of BeT compared with different baselines in learning from demonstrations. For CARLA, we measure probability of car reaching goal successfully. For Block push, we measure probability of reaching one and two blocks, and the probabilities of pushing one and two blocks to respective squares. For Kitchen, we measure the probability of n tasks being completed by the model within the allotted 280 timesteps. Evaluations
100 rollouts CARLA and 1,000 rollouts in Block push and Kitchen environments. Kitchen
CARLA
Block push
Driving Success
Reach
Push
# Tasks completed
Baselines
RI
R2
P1
3
5
RBC
0.98
0.67
0
0
0
0
1-NN
0.99
0.49
0.05
0.01
0.90
0.72
0.44
0.17
0
LWR
1
0.50
0.06
0
1
0.83
0.52
0.21
0
VAE
0
0.60
0.05
0
1
0
0
Flow
0.03
0.59
0.02
0
0.04
0
0
IBC
0.25
0,98
0.04
0.01
0
0.99
0.87
0.61
0.24
0
BeT (Ours)
0.98
1
0.99
0.96
0.71
0.99
0.93
0.71
0.44
0.02
Demonstration
Implicit
Behavior
m
100
100
200
100
200
Frequency
Thak
Task
bottom burner hinge cabiner
kettle
light switch
microwave
slide cabinet
top burner
not completed
Figure 4: Distribution of most frequent tasks completed sequence in the Kitchen environment. Each task colored differently, and frequency shown out of 1,000 unconditional rollouts from the models.
3.4 Does BeT capture the modes present in behavior data?
Next, examine the question of whether, given dataset where multi-modi behavior exists, model learns behavior that is also multi-modal. Here, we are interested sceing the variance the behavior of the model over different rollouts. In each of our environments, the demonstrations contain different types of multi-modality. As result, we show a comprehensive analysis of multi-modality seen in agent behaviors.
Table 2: Multimodality learned from the multimodal demonstrations by different algorithms. CARLA, consider the probability of turning left vs. right the intersection, ignoring OOD rollouts. Block push, we consider two set of probabilities, (a) which block was reached first, and (b) what was the pushing target for each block. Finally, in Franka Kitchen, we consider the empirical entropy for the task sequences, considered as strings, sampled from the model. We highlight the values closest the corresponding demonstration values.
Block: first
Push: red
Push: green
CARLA
Kitchen
block reached
block target
block target
Baselines
Left
Right
Red
Green
Red
Green
Red
Green
Task entropy
RBC
0.98
0.41
0.25
0
0
0
I-NN
0.99
0.24
0.25
0
0.01
2.12
LWR
0.26
0.26
0.01
0.01
0.01
2.29
VAE
0
0.27
0.33
0
0
0.72
Flow
0.31
0.29
0
0
0.08
IBC
0.12
0.13
0.48
0.50
0
0.01
0.01
2,41
BeT (Ours)
0.34
0.64
0.54
0.46
0.43
0.44
0.41
0.40
2.47
Demonstrations
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
2.96

---

Dataset
MLP MSE DC
DeT with history
BeT with history
Figure 5: Comparison between an RBC model and two BeT models, trained with and without historical context on a dataset with three distinct modes. BeT with history is better able to capture the context-dependee behavior in the demonstrations.
We see in Table that in CARLA and Block push, BeT covers all the modes of the demonstration data, even in the few cases where it does not perfectly match the demonstrated task probabilities. For the Kitchen environment, we see in Fig. 4 that BeT visits certain strings of tasks more frequently than in the original demonstrations. However, compared to other strong baselines, BeT generates longer task strings more often while maintaining diversity and not collapsing to a single mode.
3.5 How important are the individual components of BeT?
There are four key differences between BeT architecture and standard BC: (a) binning actions into discrete clusters, (b) using offsets to faithfully reconstruct actions later, (c) learning sequentially to use historical context, and (d) using an attention-based MinGPT trunk. In this section, we discuss the impacts they have BeT performance.
Impact of discrete binning: Intuitively, hav- ing discrete options for bin centers is what en-
Table 3: Relative performance of ablated variants of BeT, normalized by average BeT successes at the task
ables BeT to express multi-modal behavior even
Ablations
CARLA
Block push
Kitchen
when starting from an identical starting state.
No offsets
0,94
0.95
0.78
Indeed, if there is no binning, we see from Ta- ble 3 that the performance of BeT drops signif- icantly. More tellingly. in the Franka Kitchen environment, the model only ever completed subsequence of (kettle, top/bottom burner, light switch, slide cabinet) tasks after 100 random rollouts. This result shows that having dis- crete bins helps BeТ achieve multi-modality. We also experiment with the Mixture density networks (MDN) [8] and uniform quantization, as shown previous works [25, 39]. We see that they may perform well sometimes but overall still fall short of our k-means binning approach.
No binning
0.94
0.25
0.68
No history
0.65
0.95
0.88
MLP
0.90
0
0.05
Temp. Conv
0.72
0.01
0.26
LSTM
0.03
0.03
0.04
GPT-MDN
0,30
0.83
0.86
Unif. quant.
0.90
0.96
0.90
Necessity of action offsets: An important feature of BeT the residual action offset that corrects the discrete actions coming from the bins. While the bin centers may be quite expressive, Table 3 shows that the inability to correct them causes a performance degrade. Interestingly, the largest degradation comes in the Kitchen environment, which also has the highest dimensional action space. Intuitively, we can understand how in higher dimension the loss of fidelity from discretizing would be higher, and the relative performance loss across three environments support that hypothesis.
Importance of historical context: While RL algorithms traditionally assume environments are Markovian, human behavior in an open-ended environment is rarely so. Thus, using historical context helps BeT to perform well. We show a simple experiment in Fig. 5 on the second point mass environment. Here, training and evaluating with some historical context allows BeT to follow the demonstrations better. We experience the same in the CARLA, Block push, and Kitchen environments, where training with some historical context raises performance across the board as seen in Table 3.
Importance of transformer architecture: Despite transformers' success other fields of machine learning, it is natural to wonder whether the tasks BeT solves here really requires one. We ablated

---

BeT by replacing the MinGPT trunk with an MLP, Temporal Convolution, and LSTMs, and found that they have lower performance while also being difficult to train stably. This performance reduction remains even if the ML. is given some historical context by stacking h observations before passing it to the MIP. See Table. for results and Appendix C.2 for further details.
Ablating the number of discrete bin centers, k: Since BeT is trained with sum of focal loss for the binning head and MSE loss for the offset head, the number of cluster centers present a trade-off in the architecture. Concretely, as the number of bins go up, the log-likelihood loss goes up but the MSE loss goes down. In Table 3, we showed that using only one bin (k: = 1) decreases the performance level of BeT.
CARLA
Franka krtchen
1.40
Gest Bel
3.00
Norrel Fred reserd
0.95
reaard
0.95
0.30
0.53
0.85
0.39
0.95
0.75
8.63
0.65
61
128
256
M Mamber of brinx (k)
Ninbhw of binx (8)
Figure 6: Ablating the number of discrete bin centers for BeT. Reward is normalized with respect the best performing model.
In Fig. 6, we present the plot of the variation in performance as l value changes. We see that for BeT it matters somewhat to pick a number of clusters k that is in the right neighborhood. However, the range for near-optimum performance is quite wide. In our experiments, we also pick a in the right neighborhood and only run a sweep at the very end to find out an optimal value for k.
Computation considerations: While transformers in usual contexts are large models, we downscale them for application in BeT (See Appendix B.4). Our models contain on the order of - parameters, and even with a small batch size trains within an hour for our largest datasets (Block push) on a single desktop GPU. In contrast, for the same task, our strongest baseline IBC takes about 14 hours. Evaluation rollouts on the same environment take 1.65 seconds with Be'T, as opposed to 17.70 seconds with IBC.
Related Work
This paper builds upon a rich literature in imitation learning, offline learning, generative models, and transformer architectures. The most relevant ones to our work are discussed here.
Learning from offline data: Since Pomerleau [67] showed the possibility of driving an autonomous vehicle using offline data and a neural network, learning behavior from offline data has been continuous topic of research for scalable behavior learning [2, 7, 75J. The approaches can be divided into two broad classes: Offline RL [27, 44, 45, 80, 47, 26], focusing on learning from datasets of a mixed quality that also have reward labels; and imitation learning [61, 63, 64, 37], focusing on learning behavior from dataset of expert behavior without reward labels. BeТ falls under the second category, as it is behavior cloning model. Behavior cloning is a form of imitation learning that tries to model the action of the expert given the observation which is often used in world applications [84, 85, 84, 69, 24, 83]. As behavior cloning algorithms are generally solving a fully supervised learning problem, they tend to be faster and simpler than reinforcement learning offline RL. algorithms and in some cases show competitive results [26, 33].
Generative models for behavior learning: One approach for imitation learning Inverse Rein- forcement Learning IRL (74, 59], where given expert demonstrations, a model tries to construct the reward function. This reward function then used to generate desirable behavior. GAIL [37], an IRI. algorithm, connects generative adversarial models with imitation learning to construct model that

---

can generate expert-like behavior. Under this IRL framework, previous works have tried to predict multi-modal, multi-human trajectories [46, 38]. Similarly, other works have tried Gaussian Processes [71] for creating dynamical models for human motion [79]. Another class of algorithms learn generative action decoder [65, 52, 76] from interaction data make downstream reinforcement learning faster and easier, which inspired BeTT's action factorization. Finally, a class of algorithms, most notably [50, 25, 43, 58J do not directly learn a generative model but instead learn energy based models. These energy based models can then be sampled to generate desired behavior. Since [25] is a BC model capable of multi-modality, we compare against it as a baseline in Sec. 3.
Transformers for control: With the stellar success of transformer models [78] in natural language processing [19, 11] and computer vision [22], there has been significant interest in using transformer models to learn behavior and control. Among those, [12, 39] applies them Reinforcement Learning and Offline Reinforcement Learning, respectively, while [13, 16, 54] use them for imitation learning. [16, 54] use transformers mostly summarize historical visual context, while [13] relies their long-term extrapolation abilities to collect human-in-the in-the-loop demonstrations more efficiently. BeT is inspired by both of these use cases, while leveraging its generative abilities. Architecturally, BeT is most closely related to the imitation learning variant of [39], with significant difference that while [39] learns the joint state, action distribution, BeT learns the conditional distribution of action given state, which allows BeT to tackle much more complicated state spaces.
use a transformer
summarize historical context
Datasets for distributionally multi-modal data: Similar to computer vision [18, 48, 51J and natural language processing [10, 70], there has been a recent interest in collecting behavior datasets that may aid in downstream behavior learning. Some of them are labeled with agent goals or rewards for downstream tasks [55, 26, 56], while others are more open ended [34, 52, 82] and come without reward or task labels. In our work, we focus towards the latter class. The lack of labeled goal or reward labels in the second category implies that there more modtityy in the action distributions compared to action distributions of goal or reward conditioned datasets, which is the same reason lot of work learning from multi-modal datasets try to learn a noal-conditioned model [35, 34, 52, 16]. Finally, the lack of labelling requirements mean that the unlabelled datasets are cheaper to obtain, which should help BeT scale further in the future.
Discussions
In this work, introduce Behavior Transformers (BeT), which uses a transformer- based backbone with a discrete action mode predictor coupled with a continuous action offset corrector to model continuous actions sequences from open-ended, multi-modal demonstrations. While BeT shows promise, the truly exciting use of it would be to learn diverse behavior from human demonstrations or interactions in the real world. In parallel, extracting a particular, unimodal behavior policy from BeT during online interactions, either by distilling the model or by generating the right 'prompts' [72], would make BeT tremendously useful prior for online Reinforcement Learning.
Acknowledgments
We thank Ben Evans, David Brandfonbrener, Abitha Thankaraj, Jyo Pari, and Anthony Chen for valuable feedback and discussions. This work was supported by grants from Honda, and ONR award numbers 1-0014-21-1-24001 and N00014-21-1-2758.
10

---

References
[1] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino,
M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv: 101001191, 2019.
[2] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483,99448338088 2009.
[3] P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning- based framework for efficient dexterous manipulation. arXiv preprini arXiv:2203. 13251,, 2022.
[4] C. G. Atkeson, A. W. Moore, and S. Schaal. Locally weighted learning. Lazy learning, pages 11-73, 1997.
[5] S. Bai, J. Z. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv://03.0/27// 2018. regularization for [7] A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Survey: Robot programming by demonstra- tion. Handbook of robotics, 59(BOOK _CHO),, 2008
[6] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: 0444990 2021.
self-supervised learning. arXiv preprint arXiv:2105.0445:
[8] C. M. Bishop. Mixture density networks. Neural Computing Research Group Report, 1994.
[9] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel,
M. Monfort, U. Muller, J. Zhang, al. End to end learning for self-driving cars. arXiv preprint 1riiv:1604.07:11 2016.
[10] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 1508. 05326, 2015.
[11] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.1416 14165, 2020.
[12] I. Chen, Lu, A. Rajeswaran, Lee, A. Grover, M. Laskin, Abbeel, Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. CoRR, abs/2 10,00440, 2021. URL
http:////////a////////////// org/ abs/2106. http//106.0114555.
[13] H. M. Clever, A. Handa, H. Mazhar, K. Parker, O. Shapira, Q. Wan, Y. Narang, I. Akinola, M. Cakmak, and D. Fox. Assistive tele-op: Leveraging transformers to collect robotic task demonstrations. arXiv preprint 22 2021.
[14] F. Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9329-9338, 2019.
[15] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. GitHub Repository. 2016
[16] S. Dasari and A. Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:201 1.05970 2020.
[17] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933-941. PMLR, 2017.
[18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2009. doi: 10. 10..10/9///.009..... 200.CVP9R0009550084488888888
[19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, pages 4171-4186.
2018. doi: 10. 11863/v1/N119/ 1423.
[20] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. arXiv preprint arXiv: 1605 08803, 2016.
[21] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
11

---

[22] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[23] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning. volume 48 of JMLR Workshop and Conference Proceedings, pages 1329-1338. JMLR.org. 2016.
[24] P. Florence. L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499 2019.
(25) P. Florence, C. Lynch, A. Zeng. O. Ramirez, A. Wahid, I.. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. arXiv preprint 77Xi:2210900013: 2021.
[26] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven
reinforcement learning. arXiv preprint wwxxii:0004072//99 2020.
[27] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, PMLR, 2018.
[28] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning, pages 2243-225, PMLR, 2017.
[29] F. A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: Continual prediction with Istm. Neural computation, (2110):441124711 2000.
[30] R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer
vision, pages 1440-1448, 2015.
[31] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentrie video. arXiv preprint arXiv:21 10.00758, 3, 2021.
[32] J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
[33] C. Gulcehre. Z. Wang. A. Novikov, T. Le Paine, S. Gomez Colmenarejo, K. Zolna, R. Agarwal,
J. Merel, D. Mankowitz, C. Paduraru, et al. Rl unplugged: Benchmarks for offline reinforcement learning. arXiv e-prints, pages arXiv-2006, 2020.
[34] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 191.v11900.11956. 2019.
[35] K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. Advances in neural information processing systems, 30, 2017.
[36] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1 1.11109////..00
[37] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, volume 29, pages 4565-4574 2016.
[38] B. Ivanovic, E. Schmerling, K. Leung. and M. Pavone. Generative modeling of multimodal multi-human behavior. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3095- IEEE, 2018
[39] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
[40] N. Kalchbrenner, L. Espeholt, K. Simonyan, A. v. d. Oord, A. Graves, and K. Kavukcuoglu. 1tt.:1610010090 2016.
Neural machine translation in linear time. arXiv preprint
[41] A. Karpathy. GitHub- - karpathy/minGPT. A minimal PyTorch re-implementation of the OpenAl
GPT (Generative Pretrained Transformer) training, 08 2020. URL https: //ggt /github. com/ karpathy/ninGPr,
12

---

[42] D. P. Kingma and M. Welling. Anto-ing variational bayes. arXiv preprint arXivv1312.60 2013.
[43] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.cm 20000,0000,00000, 2021.
[44] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off oollyy q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
[45] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[46] N. Lee and K. M. Kitani. Predicting wide receiver trajectories in american football. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9. IEEE, 2016.
[47] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01645 2020.
[48] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[49] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[50] M. Liu, T. He, M. Xu, and W. Zhang. Energy-baseds imitation learning. arXiv preprint arXiv:2004.0920 09395, 33, 2020.
[51] Z. Liu, P. Luo, X. Wang, and X. Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[52] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning, pages 1113-1132. PMLR, 2020.
[53] J. MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium mathematical statistics and probability, volume 1, pages 281 2-17 Oakland, CA, USA, 1967.
[54] Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. arXiv preprint 1rXiv:2110.13423, 2021.
[55] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta,
E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879- 179-893. PMLR, 2018.
[56] A. Mandlekar, D. Xu. J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei- S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2 /08. 03298, 2021.
[57] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-555 533, 2015.
[58] O. Nachum and M. Yang. Provable representation learning for imitation with contrastive fourier features. arXiv preprint arXiv:210S. 12272, 2021.
[59] A. Y. Ng, J. Russell, et al. Algorithms for inverse reinforcement learning. In lcml, volume 1, pages 663-670, 2000.
[60] A. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O, Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.08. 03499, 2016.
[61] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.064 011/8/1006711 2018.
[62] Pari, N. Muhammad, S. Arunachalam, L. Pinto, et al. The surprising effectiveness of representation learning for visual imitation. arXiv preprint /rXiv:22/2.015// 2021.
[63] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guides deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
13

---

[64] X. B. Peng, Z. Ma. P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based based character control. ACM Transactions on Graphics (TOG), 1000))1-20, 2021.
[65] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 1944, 2020.
[66] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 119444 2020.
[67] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network, Advances in neural information processing systems, 1, 1988.
[68] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305-313, 1989.
[69] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE international conference robotics and automation (ICRA), pages 3758-3765. IEEE, 2018.
[70] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 05250, 2016.
[71] C. E. Rasmussen and H. Nickisch. Gaussian processes for machine learning (gpml) toolbox. The Journal of Machine Learning Research, 11:3011-3015 2010.
[72] L. Reynolds and K. McDonell. Prompt programmiing for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7, 2021.
[73] S. Ross, G. Gordan, and A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In arXiv preprint arXiv: 1eee:0111.000 1.0686, 2010.
[74] S. Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Compulational learning theory, pages 101-103, 1998.
[75] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3 (6):233-22)) 242, 1999.
[76] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011. 2020.
[77] Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.00 1805.01954, 2018.
[78] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 5998-6008, 2017.
[79] J. M. Wang, D. Fleet, and A. Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence, 30(2):283-295: 2007.
[80] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 191 I. 1361, 2019.
[81] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved augmenedd reinforcement learning. arXiv preprint arXiv:2107.09645. 2021.
[82] S. Young, J. Pari, P. Abbeel, and L. Pinto. Playful interactions for representation leamning arXiv preprint N.Xi::2177000444 2021.
[83] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin,
D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, 2020.
[84] T. Zhang, Z. McCarthy, Jow. D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, pages 5628-5635. IEEF, 2018.
[85] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills, arXiv preprint arXiv: 1802.08800.0954 2018.
14


Reviews:
[{"review_id": "YJ2jXwme3C2", "reviewer": "Reviewer_oVB2", "summary": "This paper proposes an imitation learning method to take into account the variance in expert demonstrations. In detail, a novel Behaviour Transformer model is proposed, which uses a k-means clustering based pre-processing approach to extract k modes from the noisy expert demonstrations. The proposed approach exploits the ability of transformer based approaches to predict complex multi-modal distributions.  The proposed model is evaluated on diverse tasks such as, CARLA, Block Pushing and Franka Kitchen. ", "questions": "In addition to the points above,\n\n* The motivation behind the choices of the baselines should be made clear? Currently the baselines chosen are quite simple.\n\n* It is unclear if the method can truly deal the variance in expert demonstrations, expect for the simplistic point mass environments?\n\n* Why does the VAE baseline fail completely on the CARLA experiments?", "limitations": "Currently the paper does not discuss the limitations of the the proposed BeT model in detail, e.g. wrt the computational resources required for training, whether the proposed method scales to large scale datasets and tasks, the suitable application areas.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths,\n* The proposed method is simple and is able to capture the variance of expert demonstrations as demonstrated in case of the point mass environments.\n* The paper is well written and easy to understand.\n* The paper includes ablations on the effect of discrete binning, action offsets and historical contexts, which shows the importance of these components.\n\n\nWeaknesses,\n\n* Effect of L2 loss in L130: Even after k-means discretization in L106, there could be considerable variance within each cluster. Using the L2 loss as in L130 would kill this variance? In case of the point mass environment in Figure 2 and 5, this loss would not have a large effect as the within cluster variance is low. The paper should include examples where the true variance of the expert demonstrations is captured.\n\n* In Figure 5, BeT with history seems to prefer the red mode over the other two modes. While in the dataset all three modes are equally likely. It is unclear, if the BeT model can capture the true underlying distribution of expert demonstrations?\n\n* Importance of the transformer model: While there is not denying the effectiveness of transform models, for the simple tasks considered here, e.g. point mass environments, it is not clear if the MinGPT model is actually necessary. It would be interesting to consider a LSTM based baseline as it would enable the modelling of historical contexts unlike MLPs considered in Table 3. \n\n* The CARLA experimental setting seems very simple. It is also unclear how the expert demonstrations are captured? The amount of variance (if any) in the expert demonstrations is unclear. This is further illustrated by the fact that RBC baseline which cannot deal with variance in the expert demonstrations works so well. Moreover, the paper should include further details like number of traffic participants in the scene, so that the reader can correctly judge the complexity of the task. \n\n* Baselines: The paper compares to only simple baselines like VAE, Flow and IBC. Prior state of the art works such as GAIL [a] are not considered as baselines.\n\n* Equation numbers are missing, e.g. L130.\n\n[a] Generative Adversarial Imitation Learning, Ho et. al. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PuCDI2ydm2X", "reviewer": "Reviewer_xCyV", "summary": "The paper presents an approach to imitation learning using transformers. The core challenge this work aims to tackle is enabling imitation learning methods to better handle multiple behavior modes in demonstration data. The key idea amounts to learning a hybrid discrete+residual action representation using k means, then using a minGPT transformer architecture to produce actions conditioned on the history of observations and actions. The experiments show strong performance on tasks with multi-modality in the demonstrations compared to baselines, importance of each of the components, and some qualitative examples highlighting the methods ability to capture multimodality.", "questions": "- Can the authors explain the differences of their approach with the imitation learning version of trajectory transformer? A comparison would also be good to see.\n- A more minor point: are there places where k-means fails is insufficient for binning the actions? I can imagine cases where small differences in the action can have huge impacts in task success, and perhaps k means may incorrectly group two functionally different actions to the same group. Perhaps there are ways of doing task-aware discretization of the actions. ", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The paper tackles an important problem in handling multi-modal behavior in demonstration data. Handling such data will be important for enabling imitation learning algorithms to scale to broader datasets. \n- The proposed approach is clean, simple, and well motivated. Indeed sequence models seem to be able to capture very multi-modal distributions in text, and by discretizing actions it makes sense that the sequence model can also capture multi-modality in behavior.\n- Experiments are also thorough, and compare the method to most of the relevant baselines (Implicit BC, Nearest Neighbors, BC) across multiple environments. The qualitative examples also nicely highlight how the method captures many modes in behavior, and how historical context is important for doing so.\n- The ablations show that changing the architecture or removing the action binning/offset both hurt performance, suggesting that all of the components are important.\n- Overall, the paper is well written.\n\n*Weaknesses*\n- The most significant weakness/question I have with this work is the novelty compared to the Trajectory Transformer (Janner et al). In the related work this paper simply states Trajectory Transformer as different because it focused on Offline RL. However, the Trajectory Transformer paper also proposes a version of the method that can be used for pure imitation learning (not just offline RL), and has corresponding experiments. Moreover, TT also tokenizes actions, and uses minGPT to generate sequences. The Trajectory Transformer paper proposes using beam search to guide actions toward certain goals, but without the beam search, the unconditional generation seems very similar to what is being proposed in BeT.  I can see some implementation differences like the masked MT loss and the k-means for discretization, but at a high level the methods appear very similar. At the very least, the current related work significantly underplays the similarities to TT, and I think a more comprehensive discussion of how the two are related is important. It should probably also be added as a comparison.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H2qy1GPqgkN", "reviewer": "Reviewer_4yjG", "summary": "This paper tackles the problem of behavioral cloning from distributionally multi-modal experience. The proposed solution is \"Behavior Transformers\" (BeT). Specifically, the authors propose clustering continuous actions into discrete bins using k-means, and model multi-modal policy as categorical distributions with a transformer-based sequence model.\n\nEmpirical evaluations are performed on five environments ranging from simple diagnostic point mass to high-dimensional robotics and self-driving environments. Comprehensive analysis is provided to prove that BeT is indeed capable of learning multi-modal behavior in these environment and better than the baseline methods chosen by the authors. Detailed ablations studies are presented to analyzed the importance of each components of the proposed methods.", "questions": "* Table 3 shows average BeT performance at the task. Can you be more specific about what they are averaged across?\n* The text in the pdf cannot be searched and selected. This makes review difficult.\n* Is this a related work? Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G. and Lim, J.J., 2017. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. Advances in neural information processing systems, 30.", "limitations": "Very little to no discussion on limitations and potential negative societal impact. Please add relevant pieces to the conclusion section. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Very clear problem statement about imitating multi-modal experience (Sec. 2) and issues with canonical BC that assumes an unimodal expert.\n* Clear empirical evidence that BeT is able to imitate multi-modal behavior (Sec. 3.3)\n* Use diverse testing environments ranging from simple point mass to high-dimensional robotics and self-driving environments.\n* Sufficient ablations that analyze the importance of describe binning, action offsets, interaction history, and transformer architecture.\n\nWeaknesses:\n* Imitating multi-modal experience is well-acknowledged problem. However, how the proposed method different from previous contributions in this line is barely discussed. Furthermore, I think there should be citation and discussion about prior efforts in introduction. Intended or not, it somewhat gives me an impression as if this is the first work tackling multimodal imitation learning. \n* Insufficient empirical comparison: The only previous contribution about multimodal imitation learning that this work compare with is \"Implicit Behavioral Cloning\" (IBC) [22]. Because of the IBC paper shows outperforming Mixture Density Network (MDN) [7], a well-established baseline for multi-modal imitation learning, the authors skipped comparing with MDN. These naturally trigger the following questions: (1) Have you compared to MDN or other approaches using mixture distributions apple-to-apple, using the same transformer and also historical context? (2) The exact same question for the comparison between BeT and IBC. Both MDN and IBC can be more easily implemented than the proposed clustering and binning and trained end-to-end. Without fair comparisons, I don't feel convinced about using the proposed method.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-ve5-Yx-37j", "reviewer": "Reviewer_uHYM", "summary": "This paper describes a method for Behavior Cloning that leverages 1) a transformer-based architecture to model the state-action relationships, thereby lifting some of the Markovian assumptions many BC architectures make, and 2) a discrete+continuous residual action representation, which enables the model to learn a multimodal action representation more easily through binning of the discrete action clusters, while not incurring a large loss of precision by explicitly learning a continuous offset from the cluster centers.", "questions": "1- I am very curious about the choice of k-means as a discretization scheme vs. e.g. uniform quantization. Given the added correction term, it is not immediately obvious that performing k-means on the action is worth the added complexity. A uniform binning would also be more general and potentially would generalize better across tasks and embodiments. I would have loved to see that particular ablation.\n2- The paper doesn't mention computational considerations at run-time, only at training time. I would have liked to see some discussion of the tradeoffs here, particularly if the model is to be used for real-time continuous control.", "limitations": "Nothing comes to mind.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the proposed architecture is well-motivated: much of recent works in BC have pointed at the weaknesses the model proposes to address, and the model architecture derives very naturally from the goals of addressing those weaknesses.\n- the experimental results compare against strong baselines, and ablation experiments clearly tease out why the model performs well against them.\n- the paper is limpid, well-organized and immediately useful to the community.\n\nWeaknesses:\n- No real-world experiment. This may not be essential in this context, but would inform how practical it is to deploy this kind of architecture in a real-time, real-world system.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept