Learning the Representation of Behavior Styles with Imitation Learning

Abstract: Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions. However, behavior styles such as motion trajectory and driving habit depend largely on the dataset of human maneuvers, and settle down to an average behavior style in most imitation learning algorithms. In this study, we propose a method named style behavior cloning (Style BC), which can not only infer the latent representation of behavior styles automatically, but also imitate different style policies from expert demonstrations. Our method is inspired by the word2vec algorithm and we construct a behavior-style to action mapping which is similar to the word-embedding to context mapping in word2vec. Empirical results on popular benchmark environments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly. Furthermore, compared with various baselines, our policy influenced by its assigned style embedding can better reproduce the expert behavior styles, especially in the complex environments or the number of the behavior styles is large.

Reviews and author responses:

The paper is about imitation learning, and it particular it considers the task of replicating the style of behaviours. The authors propose a novel framework, called Style Behavioural Cloning that can learn the representation of different behaviour styles from demonstrations by experts. The approach is an extension of behaviour cloning, that tries to derive the behaviour style embedding by optimising the model to predict the action accurately.
The paper presents the mathematical formulation of the framework and an experimental analysis of different settings. + Extending imitation learning frameworks taking into account the behavioural style is a very interesting topic with a potential of application in a variety of domain, from autonomous driving to HRI

- From the reading, several aspects are not completely clear to me. I am not an expert on the problem, and I might have missed or misunderstood something, but I will try to summarise the concerts below:

   * Not completely clear what do the authors mean saying that, with respect to behavioural cloning, their method tries to solve the problem in    
     a more direct way (introduction)
   * Not clear if style MDP is introduced by the authors or already existing in literature. Also I do not understand how the assumption of having 
     a behaviour policy as mixture of behaviour styles is reflected in the formulation (2.2)
   * Not clear how the authors select the style vector c for each episode (3.1)
   * What is in essence the role of the shared layers and the policy layers? (3.1). Overall, I find Fig. 2 a bit complicated, and not very well 
     contextualised with respect to the text (it does not always clarify as it should do)
   * As the task of imitation is contextualised to an action classification task, is there a test set to be used? As the plots in Fig. 4 are on a 
     training set, they only provide a partial view of the picture and does speak about other important properties of the model, as the stability 
     and generalisation for instance (4.1.2)
   * Although the experimental analysis includes different settings, I find it a bit limited

As a general comment, in some parts the language should be revised as it is a bit difficult to follow.
 Although I find the task very interesting and with a wide potential of application, I have a few concerns about the clarity of the paper.

The paper proposes a method to learn different behavior styles from expert demonstrations via imitation learning. To achieve this, the method learns a latent style embedding of behavior styles and condition the policy on it. A style policy that takes the style embedding as additional input is learned alongside a target policy without the style embedding. Both policies are trained with behavior cloning (BC). Experiments are performed on simple environments such as Grid-World and Cartpole. **Strength:**

- Learning different behavior styles could be important for imitation learning to resolve ambiguity when the agent can choose different behaviors in the same state.

**Weakness:**

- The paper is not well written, and there are numerous grammar errors and typos (see detailed comments). The technical section also lacks motivation for each design, making it hard to follow.
- The idea of the proposed approach also has little novelty. [1] already studied how to learn a latent representation with VAE and use it for behavior cloning (BC) and GAIL. There is also no motivation about why the proposed idea might be better compared to VAE-based approaches for behavior style learning such as [1].
- The experiments are only performed using very simple environments such as Grid-World, Cart-Pole, and Highway-Env.  In these environments, the method is only compared with all baselines on Cart-Pole. In contrast, similar work [1] has performed experiments with more difficult tasks such as human motion imitation with humanoids, which has high dimensional state and action spaces. Experiments on more and harder MuJoCo tasks as well as comparisons with all baselines need to be provided to demonstrate the method's effectiveness.
- There are no ablation studies justifying the design of the method, including the use of the target policy, shared layers, and soft update in Eq (5).
- The method requires selecting the number of behavior styles beforehand, which could be limiting when the number doesn't correspond to the actual number of styles. This again raises the question of why not just use VAE-based approach such as [1].

[1] Robust Imitation of Diverse Behaviors. Wang et al., NeurIPS 2017.

**Detailed Comments:**

Just to name a few grammar errors and typos:

- "it may not always generate" → "they may not always generate"
- "help the model predicting" → "help the model predict"
- "We use the target policy here because we take the outputs of the shared layers of the target network as a high level representation of the state and copy it to the style policy can make the optimization process of the style policy faster and more stable." grammar error In summary, the proposed idea lacks novelty and is similar to prior work [1]. Experiments are only performed on simple environments without proper comparison with all the baselines. There are also no ablation studies to understand the design of the method better. Thus, I vote to reject this paper.

This paper presents Style BC, an extension to standard behavior cloning that uses an additional style embedding to capture different modes in expert demonstrations. The style embeddings are discrete, fixed-size vectors which the policy uses to predict actions from states. Style BC trains two imitative policies: the target policy and the style policy. The target policy directly predicts actions from states; the style policy additionally takes as input the selected style embedding. The style embeddings are trained by back-propagating action prediction gradients. Style BC outperforms the selected baselines (BC, GAIL, InfoGAIL) on both diversity and final performance in the CartPole environment. It is shown to be able to capture different modes from expert demonstrations in the Highway environment.

 === Strengths ===

+ The paper is well written and the proposed idea is conveyed very clearly.
+ The proposed method, StyleBC, is simple yet effective as it outperformed the selected strong baselines such as Info-GAIL in the CartPole environment. This is a plus. I am also very surprised that simply training the embedding with the highest prediction score works so well.

=== Weaknesses ===

Method
- I am not in favor of using a fixed sized (N) set embedding vectors, as this requires domain knowledge of the task.
- My main concern with the proposed method is its core idea of using the style embeddings. Since it already uses a discrete set (N) of embedding vectors, why not simply use a conditional architecture, similar to Codevilla [1]? I would love to see experimental evidence if authors believe using discrete embedding is better than conditional branching.

Experiments
- There is no ablation study at all. I would like to see some ablation experiments that justify the design choice of StyleBC. For example, how would directly training the style policy compare to using a separate target policy for the state representation?
- I wonder why there is no entry for Burn-InfoGAIL in table 1, as it is discussed in the related work that Burn-InfoGAIL should be preferred over InfoGAIL.
There is no comparison of StyleBC with baselines for the highway environment. There is no comparison with strong baselines in the gridworld environment.

[1] End-to-end driving via conditional imitation learning, Codevilla et al., 2018
 The paper is well written and the proposed idea is well conveyed. However, there is no ablation study of the design choices, and overall experiments are not thorough. I do not recommend acceptance at this moment.

This paper focuses on learning different behavior styles from expert demonstrations for imitation learning. More specifically, the contributions are three folds:
1. This paper proposes a novel style behavior cloning approach that learns the various representation of behavior styles from expert demonstrations;
2.  The style embedding which can provide valuable guidance for the generation of style policies is obtained automatically;
3.  The authors empirically demonstrate the effectiveness of the proposed method on a Grid-World environment, cartpole control tasks from OpenAI gym, and a collection of environments for decision-making in autonomous driving named highway-env. Strengths:
1. This paper is generally well written and easy to follow. The motivation of learning multiple different behavior styles is clearly stated with sufficient background information. The proposed method is validated based on multiple benchmark environments and obtains supportive experimental results.

Weaknesses:
1. In the proposed method, the number of behavior styles is a hyperparameter and the behavior styles need to be pre-defined explicitly based on human prior knowledge. However, in many tasks/scenarios, the behavior styles are hard to define and may even be impossible to be separated into distinct classes. It would be better to discuss the potential generalization to such tasks or the limitations of the proposed method.
2. In the grid-world environment, what is the value of N? Is it the same as the true number of behavior styles (i.e., 25)? In the testing policies, why do all the policies make turning immediately after passing the door, is it because there is no other turning behavior at other columns in the expert demonstrations?
3. Following up on the last question, is the proposed method able to explore/learn the feasible behavior styles not present in the expert demonstrations?
4. The current video demo has no explanation, which is hard to understand the differences between the three behavior styles. It would be better to add some descriptions in the video. Overall, the idea of learning different behavior styles from expert demonstrations is interesting. However, I would recommend the authors to address my questions in the rebuttal.

Meta-review: All reviewers suggested rejection of the paper. This is based on concerns regarding novelty of results, clarity of presentation, simplicity of conducted experiments and missing ablation studies (and several other points raised in the reviews). The authors also did not submit a rebuttal. Hence I am recommending rejection of the paper.

Decision: Reject