Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an imitation learning method to take into account the variance in expert demonstrations. In detail, a novel Behaviour Transformer model is proposed, which uses a k-means clustering based pre-processing approach to extract k modes from the noisy expert demonstrations. The proposed approach exploits the ability of transformer based approaches to predict complex multi-modal distributions.  The proposed model is evaluated on diverse tasks such as, CARLA, Block Pushing and Franka Kitchen. 
strengths_and_weaknesses: Strengths,
* The proposed method is simple and is able to capture the variance of expert demonstrations as demonstrated in case of the point mass environments.
* The paper is well written and easy to understand.
* The paper includes ablations on the effect of discrete binning, action offsets and historical contexts, which shows the importance of these components.


Weaknesses,

* Effect of L2 loss in L130: Even after k-means discretization in L106, there could be considerable variance within each cluster. Using the L2 loss as in L130 would kill this variance? In case of the point mass environment in Figure 2 and 5, this loss would not have a large effect as the within cluster variance is low. The paper should include examples where the true variance of the expert demonstrations is captured.

* In Figure 5, BeT with history seems to prefer the red mode over the other two modes. While in the dataset all three modes are equally likely. It is unclear, if the BeT model can capture the true underlying distribution of expert demonstrations?

* Importance of the transformer model: While there is not denying the effectiveness of transform models, for the simple tasks considered here, e.g. point mass environments, it is not clear if the MinGPT model is actually necessary. It would be interesting to consider a LSTM based baseline as it would enable the modelling of historical contexts unlike MLPs considered in Table 3. 

* The CARLA experimental setting seems very simple. It is also unclear how the expert demonstrations are captured? The amount of variance (if any) in the expert demonstrations is unclear. This is further illustrated by the fact that RBC baseline which cannot deal with variance in the expert demonstrations works so well. Moreover, the paper should include further details like number of traffic participants in the scene, so that the reader can correctly judge the complexity of the task. 

* Baselines: The paper compares to only simple baselines like VAE, Flow and IBC. Prior state of the art works such as GAIL [a] are not considered as baselines.

* Equation numbers are missing, e.g. L130.

[a] Generative Adversarial Imitation Learning, Ho et. al. 
questions: In addition to the points above,

* The motivation behind the choices of the baselines should be made clear? Currently the baselines chosen are quite simple.

* It is unclear if the method can truly deal the variance in expert demonstrations, expect for the simplistic point mass environments?

* Why does the VAE baseline fail completely on the CARLA experiments?
limitations: Currently the paper does not discuss the limitations of the the proposed BeT model in detail, e.g. wrt the computational resources required for training, whether the proposed method scales to large scale datasets and tasks, the suitable application areas.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of behavioral cloning from distributionally multi-modal experience. The proposed solution is "Behavior Transformers" (BeT). Specifically, the authors propose clustering continuous actions into discrete bins using k-means, and model multi-modal policy as categorical distributions with a transformer-based sequence model.

Empirical evaluations are performed on five environments ranging from simple diagnostic point mass to high-dimensional robotics and self-driving environments. Comprehensive analysis is provided to prove that BeT is indeed capable of learning multi-modal behavior in these environment and better than the baseline methods chosen by the authors. Detailed ablations studies are presented to analyzed the importance of each components of the proposed methods.
strengths_and_weaknesses: Strengths
* Very clear problem statement about imitating multi-modal experience (Sec. 2) and issues with canonical BC that assumes an unimodal expert.
* Clear empirical evidence that BeT is able to imitate multi-modal behavior (Sec. 3.3)
* Use diverse testing environments ranging from simple point mass to high-dimensional robotics and self-driving environments.
* Sufficient ablations that analyze the importance of describe binning, action offsets, interaction history, and transformer architecture.

Weaknesses:
* Imitating multi-modal experience is well-acknowledged problem. However, how the proposed method different from previous contributions in this line is barely discussed. Furthermore, I think there should be citation and discussion about prior efforts in introduction. Intended or not, it somewhat gives me an impression as if this is the first work tackling multimodal imitation learning. 
* Insufficient empirical comparison: The only previous contribution about multimodal imitation learning that this work compare with is "Implicit Behavioral Cloning" (IBC) [22]. Because of the IBC paper shows outperforming Mixture Density Network (MDN) [7], a well-established baseline for multi-modal imitation learning, the authors skipped comparing with MDN. These naturally trigger the following questions: (1) Have you compared to MDN or other approaches using mixture distributions apple-to-apple, using the same transformer and also historical context? (2) The exact same question for the comparison between BeT and IBC. Both MDN and IBC can be more easily implemented than the proposed clustering and binning and trained end-to-end. Without fair comparisons, I don't feel convinced about using the proposed method.



questions: * Table 3 shows average BeT performance at the task. Can you be more specific about what they are averaged across?
* The text in the pdf cannot be searched and selected. This makes review difficult.
* Is this a related work? Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G. and Lim, J.J., 2017. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. Advances in neural information processing systems, 30.
limitations: Very little to no discussion on limitations and potential negative societal impact. Please add relevant pieces to the conclusion section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents an approach to imitation learning using transformers. The core challenge this work aims to tackle is enabling imitation learning methods to better handle multiple behavior modes in demonstration data. The key idea amounts to learning a hybrid discrete+residual action representation using k means, then using a minGPT transformer architecture to produce actions conditioned on the history of observations and actions. The experiments show strong performance on tasks with multi-modality in the demonstrations compared to baselines, importance of each of the components, and some qualitative examples highlighting the methods ability to capture multimodality.
strengths_and_weaknesses: *Strengths*
- The paper tackles an important problem in handling multi-modal behavior in demonstration data. Handling such data will be important for enabling imitation learning algorithms to scale to broader datasets. 
- The proposed approach is clean, simple, and well motivated. Indeed sequence models seem to be able to capture very multi-modal distributions in text, and by discretizing actions it makes sense that the sequence model can also capture multi-modality in behavior.
- Experiments are also thorough, and compare the method to most of the relevant baselines (Implicit BC, Nearest Neighbors, BC) across multiple environments. The qualitative examples also nicely highlight how the method captures many modes in behavior, and how historical context is important for doing so.
- The ablations show that changing the architecture or removing the action binning/offset both hurt performance, suggesting that all of the components are important.
- Overall, the paper is well written.

*Weaknesses*
- The most significant weakness/question I have with this work is the novelty compared to the Trajectory Transformer (Janner et al). In the related work this paper simply states Trajectory Transformer as different because it focused on Offline RL. However, the Trajectory Transformer paper also proposes a version of the method that can be used for pure imitation learning (not just offline RL), and has corresponding experiments. Moreover, TT also tokenizes actions, and uses minGPT to generate sequences. The Trajectory Transformer paper proposes using beam search to guide actions toward certain goals, but without the beam search, the unconditional generation seems very similar to what is being proposed in BeT.  I can see some implementation differences like the masked MT loss and the k-means for discretization, but at a high level the methods appear very similar. At the very least, the current related work significantly underplays the similarities to TT, and I think a more comprehensive discussion of how the two are related is important. It should probably also be added as a comparison.

questions: - Can the authors explain the differences of their approach with the imitation learning version of trajectory transformer? A comparison would also be good to see.
- A more minor point: are there places where k-means fails is insufficient for binning the actions? I can imagine cases where small differences in the action can have huge impacts in task success, and perhaps k means may incorrectly group two functionally different actions to the same group. Perhaps there are ways of doing task-aware discretization of the actions. 
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes a method for Behavior Cloning that leverages 1) a transformer-based architecture to model the state-action relationships, thereby lifting some of the Markovian assumptions many BC architectures make, and 2) a discrete+continuous residual action representation, which enables the model to learn a multimodal action representation more easily through binning of the discrete action clusters, while not incurring a large loss of precision by explicitly learning a continuous offset from the cluster centers.
strengths_and_weaknesses: Strengths:
- the proposed architecture is well-motivated: much of recent works in BC have pointed at the weaknesses the model proposes to address, and the model architecture derives very naturally from the goals of addressing those weaknesses.
- the experimental results compare against strong baselines, and ablation experiments clearly tease out why the model performs well against them.
- the paper is limpid, well-organized and immediately useful to the community.

Weaknesses:
- No real-world experiment. This may not be essential in this context, but would inform how practical it is to deploy this kind of architecture in a real-time, real-world system.
questions: 1- I am very curious about the choice of k-means as a discretization scheme vs. e.g. uniform quantization. Given the added correction term, it is not immediately obvious that performing k-means on the action is worth the added complexity. A uniform binning would also be more general and potentially would generalize better across tasks and embodiments. I would have loved to see that particular ablation.
2- The paper doesn't mention computational considerations at run-time, only at training time. I would have liked to see some discussion of the tradeoffs here, particularly if the model is to be used for real-time continuous control.
limitations: Nothing comes to mind.
review_title: null

Meta-review: metareview: *Summary*

The paper addresses the problem of learning from expert demonstrations, focusing on the setting where the demonstrations are pre-collected, rewards are absent, and the distribution of demonstration trajectories contains multiple modes (limiting the performance of behavior cloning). The proposed approach uses k-means to cluster continuous actions into discrete tokens which are modeled using a transformer architecture with an additional continuous offset from the cluster centers. 

*Reviews*

The discussion has resolved all reviewer concerns and strengthened the paper with additional baselines, ablations, and positioning relative to related work. All four reviewers are in agreement that this approach is novel, well justified, effective in multiple empirical settings, and that the ablation studies clearly establish why the model performs well. The final reviewer ratings are 6 (WA), 6 (WA), 7 (A) and 8 (SA). At least two of the reviewers see this paper as high impact, and I agree. I therefore recommend this submission can be accepted and considered for an outstanding paper award.

*Potential Impact*

In generative image modeling, the shift to modeling the continuous space of pixels using discrete tokens and transformers helped underpin recent massive improvements in quality (e.g., in the [DALL-E](https://arxiv.org/pdf/2102.12092.pdf) paper and others). In the image domain, the two-stage approach (learning a codebook, then training a transformer) seems to successfully capture both high-frequency details and low-frequency structure. Therefore it's interesting to see this paper apply similar ideas to the modeling of behavior/actions. To my knowledge this is the first paper to open up this direction and it could lead to further advancements with the application of more advanced clustering / codebook learning techniques and so on (as has occurred in the image domain). Therefore I see the potential impact of this paper as high.
confidence: Certain
Decision: Accept