Year: 2021

Abstract:
Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions. However, behavior styles such as motion trajectory and driving habit depend largely on the dataset of human maneuvers, and settle down to an average behavior style in most imitation learning algorithms. In this study, we propose a method named style behavior cloning (Style BC), which can not only infer the latent representation of behavior styles automatically, but also imitate different style policies from expert demonstrations. Our method is inspired by the word2vec algorithm and we construct a behavior-style to action mapping which is similar to the word-embedding to context mapping in word2vec. Empirical results on popular benchmark environments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly. Furthermore, compared with various baselines, our policy influenced by its assigned style embedding can better reproduce the expert behavior styles, especially in the complex environments or the number of the behavior styles is large.

text: Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions. However, behavior styles such as motion trajectory and driving habit depend largely on the dataset of human maneuvers, and settle down to an average behavior style in most imitation learning algorithms. In this study, we propose a method named style behavior cloning (Style BC), which can not only infer the latent representation of behavior styles automatically, but also imitate different style policies from expert demonstrations. Our method is inspired by the word2vec algorithm and we construct a behavior-style to action mapping which is similar to the word-embedding to context mapping in word2vec. Empirical results on popular benchmark environments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly. Furthermore, compared with various baselines, our policy influenced by its assigned style embedding can better reproduce the expert behavior styles, especially in the complex environments or the number of the behavior styles is large.

heading: 1 INTRODUCTION
text: Deep reinforcement learning (DRL), which combines reinforcement learning and deep neural networks, has achieved rapid development in the last few years. Research institutions as well as industrial companies have applied this technology to many domains, such as games (Atari games and Go) (Silver et al., 2016; Van Hasselt et al., 2016), robotics control (Zhang et al., 2015) and autonomous driving (González et al., 2016). However, as the DRL algorithms optimize the models according to the reward function, it may not always generate desirable behaviors due to the inferior local convergence or inappropriate reward design, especially in the real world problems where the reward function is difficult to clearly define.
Imitation learning (IL) methods have the potential to cover this shortage, with which an agent learns to perform a task from expert demonstrations by mimicking a mapping between observations and actions instead of maximizing the accumulated reward (Osa et al., 2018). In these methods, the behavior style such as the motion trajectory and operation habit is determined depending on the dataset of expert demonstrations. In practise, the real world problems such as self-driving vehicles (Munoz et al., 2009), assistive robots (Ikemoto et al., 2012) and human-computer interaction (Ross & Bagnell, 2010) may need a large amount of demonstrations and these demonstrations are usually collected from multiple human experts who may have different behavior styles. For the conventional IL algorithms, the imitated behavior tends to converge into an average of the multiple behavior styles in the demonstrations when the imitation model is trained. However, demonstrations from different experts can show significant variability because different human experts typically employ different policies according to their own styles and habits, and learning an average behavior style maybe unacceptable. For instance, in a driving scenario with a leading vehicle, aggressive drivers may make the decision to change lanes. While some drivers are relatively conservative and will maintain a stable following distance with the front vehicle by controlling the velocity, as illustrated in Figure 1. In these situations, the average behavior reproduced by imitation learning may not express any kind of expert styles, or even worse results in an non-convergent policy (e.g., crashing the leading car in the scenario of Figure 1).
There are some algorithms considering behavior style in imitation learning. A classical way to learn from demonstration is Dynamic Bayesian Network (Eppner et al., 2009), which does not use deep learning. Another way is the constrained methods, such as Probabilistic Principle Component Analysis (Perico et al., 2019), which can take into account additional constraints such as speed limits and obstacles. Although these algorithms can reproduce the behavior style from a small number of demonstrations, due to a number of assumptions about the structure of sensory and control signals, such traditional methods are not scalable to more multi-dimensional and multi-modal systems, such as systems involving tactile sensors and raw images. Alternatively, methods based on Generative Adversarial Imitation Learning (GAIL) combine supervised and reinforcement learning (Ho & Ermon, 2016), which produces similar behaviors to the expert demonstrations in a simulation environment. Based on this, Information Maximizing GAIL (Info-GAIL) (Li et al., 2017), which can embed differences in behavior style in latent space using mutual information, has been used to produce different driver models in an autonomous driving environment. However, InfoGAIL involves sampling a latent code at the beginning of each trial. While when the simulated surrounding vehicles are initialized with the real driving environment information (such as velocity and position), the random sampling of latent codes may not ensure consistency between the policy’s subsequent actions and the driver’s behaviors. To address this limitation, a Burn-InfoGAIL algorithm (Kuefler & Kochenderfer, 2017), built on Info-GAIL, draws latent features directly from a learned, inference distribution. In addition, this burn-in policy must take over from the point at which a specific expert demonstration ends.
Building upon the above researches, this paper proposes a novel imitation learning framework named Style Behavior Clone (Style BC), which can learn the representation of various behavior styles from expert demonstrations. Not only can the model automatically learn a style embedding that is meaningful to represent the behavior styles, but also accurately reproduces different expert behaviors. Our approach is an extension of behavior cloning (BC) (Bojarski et al., 2016) and different from current style involved imitation learning algorithms which try to infer the style embeddings through inputting the behavior sequences, our method try to solve this problem in a more direct way: the correct embeddings should improve the final performance of BC and thus designing a framework which can adjust the style embeddings and the policy network simultaneously according the final training performance.
To be specific, this paper introduces a hypothesis that each policy has a latent feature c, i.e. style embedding, which can determine its behavior style. Therefore, in order to improve the prediction accuracy of the actions belonging to different styles, we need an accurate c to guide the behavior. Our method is inspired by the word2vec (Mikolov et al., 2013) algorithm which is used to embed the words in natural language processing (NLP). The word2vec algorithm try to compute the word embedding through optimizing the model to predict the contexts accurately, and correspondingly, Style BC tries to compute the behavior style embedding through optimize the model to predict the action accurately. Finally, it is worth noting that our algorithm updates style embedding by making the distinction of behaviors more clear, while Info-GAIL and related deformations disentangle the latent c by inferring from the sequences of state-actions pairs.
The main contributions of this paper are summarized as follows: (i) we propose a novel style behavior cloning approach that learns the various representation of behavior styles from expert demonstrations; (ii) the style embedding which can provide valuable guidance for the generation of style policies is obtained automatically; (iii) finally, we empirically demonstrate the effectiveness of our research on a Grid-World environment, cartpole control tasks from OpenAI gym, and a collection
of environment for decision-making in autonomous driving named highway-env. We show that our model can learn to represent different kinds of behavior styles by exploring the style embedding.

heading: 2 PROBLEM FORMULATION
text: In this section, we start with a brief introduction to the background including Markov Decision Process (MDP) and style Markov decision process (style MDP). Then we provide a basic objective function of behavior cloning to learn human behaviors from demonstrations. However, due to the expert demonstrations may collect a variety of behaviors, the simple supervised policy from offline demonstration may not have the ability to reproduce these motion styles.

heading: 2.1 BACKGROUND
text: RL problems are often modeled as Markov Decision Processes (MDPs) which are defined as a tuple M = (S,A, P,R, γ). S and A represent states and actions respectively. P is a state transition probability function, which can also be represented as P (s′|s, a). Further R : S × A → R is a reward function measuring the performance of agents and γ is a discount factor for future rewards. Given an MDPM, the goal of the agent is to maximize the expectation of the discounted accumulative reward JR = E [ ∑∞ t=0 γ
tr(st, at, st+1)] automatically by finding an optimal policy π∗. In this paper, we assume that the behavior policy is a mixture of behavior styles∏ E= {πB1 (.|s, c1) ,πB2 (.|s, c2) . . . πBn (.|s, cn) }, where c is a style embedding that can determine the behavior style of policy. The objective is to reproduce a policy π(a|s, c) as an approximation of πBi , and we use the tuple (S,A,C, P,R, γ) to define this process, named style MDP.

heading: 2.2 BEHAVIORAL CLONING
text: BC as a representative imitation learning method seeks the best policy that can minimize the action prediction error in demonstrations. Traditional BC method learns a policy through supervised learning and the only requirements are pairs of input sensory observations associated with expert actions. The objective is to minimize the distribution discrepancy between the expert demonstrations and the policy, which can be formulated as follows:
min π
D [ρπ(·|s) || ρE(·|s)] , (1)
where s represents the environment state and D is the discrepancy measure like KL divergence which is a common objective function in imitation learning. In addition, ρπ and ρE depict the state action distribution generated by policy π and human demonstration E, respectively.
Human demonstrations can show significant variability because these demonstrations might be collected from users with different skills and habits. Since the policy is tied up to the demonstration in this objective function, in training the BC tends to converge into an average of various human demonstrations or a fixed behavior style when some behavior style dominates the overall demonstrations. However, sometimes we are more concerned with how to reproduce the variation of human behavior styles.
Therefore, inspired by style MDP, we try to improve the prediction accuracy of the model by learning the style embedding automatically, and leverage this encode to explore the variation of behavior styles.

heading: 3 PROPOSED METHOD
text: To learn the style embedding that captures the semantics of human behavior, our main insight is that the representation of styles can be reflected by their motions on the environment. In this paper, we assume that different style episodes in expert demonstrations are generated by a specific style policy πB (a|s, c). And what Style BC tries to do is to find the right behavior style embedding c for each episode in demonstration as well as learn a policy that can output the same action conditioning
on the that c and the state as πB (a|s, c) does. Figure 2 illustrates the overall framework of our method, which can be divided into two parts: learning the behavior style embedding and learning the imitation policy based on that embedding. The first part is implemented through the backward propagation process in the bottom of Figure 2. And for the learning of the imitation policy based on the embedding vector, we introduce an additional target policy to learn the representation of the state in the demonstration, which can also make the optimization process of the imitation policy more stable. An outline for the Style BC is shown in Algorithm 1..

heading: 3.1 THE LEARNING OF THE BEHAVIOR STYLE EMBEDDING
text: Firstly, in order to initialize the behavior style vectors C, we have to determine the number of the behavior styles N before training. We can determine it in two ways: 1) if the number of the episodes in the demonstrations is not large, we can set N to be the number of the episodes in the demonstrations; 2) otherwise if the number of the episodes in the demonstration is large, we can determine a proper value of N according to human domain knowledge, and we will introduce a method to select a right behavior embedding for each episode in the demonstrations Section 3.3. We take N as a hyper-parameter and this is the only domain knowledge related parameter in our framework.
After selecting the style vector c for each episode, we have to optimize this vector to help the model predicting the actions in this episodes more accurately. We leverage the widely used supervised
Algorithm 1: Style BC Input: N : the number of the predetermined styles, E: the expert demonstration set, θtarget and
ωtarget: the network weights of the shared layers and the policy layers in target network, θstyle and ωstyle: the network weights of the shared layers and the policy layers in style network
1 Initialize θtarget, ωtarget, θstyle and ωstyle, random initialize N behavior style vectors. 2 Select an expert episode e from E, and determine the style vector of this episode according to
Eq. 6 3 Update the target network on e according to Eq. 4 4 Update the style vector c and the policy layers ωstyle of style policy according to Eq. 2 and Eq.
3. 5 Update the shared layers θstyle of style policy according to Eq. 5
learning process for this purpose. As illustrated in Figure 2, the network named style policy is responsible for this process and it can be divided into two parts: the shared layers parameterized with θstyle and the policy layers parameterized with ωstyle . We introduce the imitation learning loss to update c and the optimization can be formulated as:
min c
D [ π(θstyle,ωstyle)(·|s, c) || πB (·|s, c) ] , (2)
where πB (·|s, c) is the action distribution calculated from the episode e with behavior style vector c.

heading: 3.2 THE LEARNING OF THE IMITATION POLICY
text: Next, we will introduce the process of optimizing the imitation policy, which is denoted as style policy in Figure 2. As mentioned above, the two network modules of it named shared layers and policy layers are parameterized with θstyle, and ωstyle respectively. For θstyle, the optimization process is similar to the process in optimizing the behavior style embedding c, which can be formulated as:
min ωstyle
D [ π(θstyle,ωstyle)(·|s, c) || πB (·|s, c) ] , (3)
This optimization is the same as in the common behavior clone algorithm and it is intuitive to introduce it here as we want the style policy to mimic the expert policy. For θstyle, we introduce a target policy to update it, the target policy is a common behavior clone network with the same network construction as the style policy, except for that its input includes the state only and will not contain the style embedding. The shared layers and the policy layers of it are paremeterized with θtarget and ωtarget respectively. In training, the parameters of the shared layers in style policy are copied from the ones in target policy. We use the target policy here because we take the outputs of the shared layers of the target network as a high level representation of the state and copy it to the style policy can make the optimization process of the style policy faster and more stable. The optimization process of θstyle, θtarget and ωtarget can be formatted as:
min (θtarget,ωtarget)
D [ π(θtarget,ωtarget)(·|s) || πB (·|s) ] , (4)
θstyle = k ∗ θstyle + (1− k) ∗ θtarget (5) where k is the hyper-parameter used in the soft update of θstyle, and π(θtarget,ωtarget) is the target policy network parameterized by θtarget and ωtarget. Note that the input of the target policy network in Eq. 4 is only the state s, which is different from the input (including both the state s and the behavior style vector c) of the style policy network in Eq. 3

heading: 3.3 SELECTING THE BEHAVIOR STYLE VECTOR FOR EACH EPISODE
text: In this section, we will explain the method for selecting the behavior style vector when the number of the episodes is large and thus we have to predetermined the number of the behavior styles before training. After selecting an episode from the expert demonstrations, we will compute the prediction accuracy of the actions in this demonstrated episode with the style policy based on each behavior style vector c in C, and select the one with the highest prediction accuracy as the behavior style vector of this episode. To be specific, the behavior style vector ce for episode e is determined through Eq. 6, where Pπ(θstyle,ωstyle)(e|c) is the prediction accuracy of style policy on the data in episode e taking the behavior style vector c as input.
ce = argmax c∈C Pπ(θstyle,ωstyle)(e|c) (6)

heading: 4 EXPERIMENTS
text: In this section, we aim at investigating (i) whether Style BC can reproduce the different behavior styles from expert demonstrations, and (ii) whether style embedding can guide and determine the generation of a specific style policy. To evaluate our approach, we conduct extensive experiments on several widely used environments: a navigation task in Grid-World, one classic control environment in OpenAI Gym (Kumar, 2020), and finally, a minimalist environment for decision-making in autonomous driving named highway-env (Leurent, 2018).

heading: 4.1 GRID-WORLD

heading: 4.1.1 EXPERIMENTAL SETUP
text: We use a 15*15 Grid-World environment as shown in Figure 3, the agent (yellow point) starts from the bottom left corner and is required to arrive at any of the five goal grid (green squares) placed at the right sideline (+20 reward). Besides, there is a wall (black squares) in the vertical direction and the agent can only pass through it from the five doors. We collected trajectories passing through different doors and reaching different five goals, as shown in Figure 3. Based on this, we conduct a set of experiments and intend to verify the following issues: (i) Whether the accuracy of behavior cloning can be improved after introducing style embedding? (ii) Whether the proposed Style BC method can reproduce different style policies? (iii) Is there an exact style embedding vector that can determine a unique style policy?
For the first issue, by comparing Style BC and traditional BC method in the above environment, we observe the differences in accuracy and reward. On this foundation, we introduce two popular imi-
tation learning algorithms, Gail and info-GAIL, and count the number of behavior styles displayed by these four algorithms. As for the last issue, several explicit style embedding is given to learn whether different styles of paths are produced.

heading: 4.1.2 EXPERIMENTAL ANALYSIS
text: First of all, from the results in Figure 4, we can find that the accuracy of Style BC is much higher than traditional BC, which proves that the introduction of style embedding will improve the prediction accuracy of behavior cloning. Moreover, the advantage of adopting style embedding is also reflected in reward. In the later stage of training, the reward of Style BC tends to be stable. On the contrary, the traditional BC’s reward fluctuates greatly because this method is sensitive to prediction errors. Figure 5 shows the number of behavior styles displayed by the four algorithms. The demonstration path is a collection of trajectories that pass through five gates and reach different goals, as shown in Figure 3. Therefore, we consider that there are 25 motion strategies in this expert demonstration. Style BC method proposed in this paper reproduces 22 different motion paths, which is the most among the four comparison algorithms; Gail and BC finally converge to only one trajectory, which reflects that the traditional imitation learning algorithm may not be able to show a variety of behavior styles; Info-GAIL represents four different motion trajectories, while we find that some different latent factors correspond to the same behavior style, resulting in only four style paths. In particular, although our method does not completely reproduce the 25 style trajectories, the number of styles reflected is much higher than the four in info-GAIL. Finally, we randomly select three vectors from the style embedding dataset generated by automatic learning as the test style. By interacting with the Grid-world environment respectively, three different motion trajectories can be obtained, as shown in Figure 6. Therefore, under the determined style embedding, only one style of behavior policy can be generated.

heading: 4.2 CARTPOLE

heading: 4.2.1 EXPERIMENT SETUP
text: This set of experiments are based on the classic control tasks named CartPole-v1 in OpenAI Gym. In this research, we assume that keeping the pendulum upright and finally staying at different specific positions (0, ±0.5, ±1.5 distance to center in CartPole) represents various behavior styles, shown in Figure 7. The expert demonstrations collect the trajectories where the end point stays at these five points. In the following experiments, Style BC is trained with the demonstrations and is hoped to reproduce different behavior styles. Similarly, we also introduce BC, GAIL and info-GAIL as the comparison algorithms.

heading: 4.2.2 EXPERIMENTAL ANALYSIS
text: The results of CartPole-v1 are demonstrated in Table 1. It’s obvious that Style BC and info-GAIL are much higher in style diversity than the other two models. BC learns the behavior style with the end point near -1.5 (-1.46), while algorithm GAIL generates a path with the end stable point near 0 (-0.02). Both behavior styles are from expert demonstrations. Furthermore, we can see that the
trajectory end point shown by Style BC is stable at 5 different positons, of which four positions stay in our expected position range. This shows that the Style BC model can replicate the four style representations shown by expert demonstrations. In particular, the end point of one of the tracks generated by Style BC is stable at 0.25. And we believe that the stable point does not fall in the desirable position (0.5) due to the deviation of style embedding. As for Info-GAIL model, although the end of the trajectories it produces also falls in five different positions, these end stable points are inconsistent with the expected position. This does not meet the requirements of reproducing different behavior styles.

heading: 4.3 HIGHWAY-ENV

heading: 4.3.1 EXPERIMENT SETUP
text: To demonstrate the effectiveness of Style BC in complex driving environment, the last experiment is carried out on Highway-env, which is a minimalist environment for decision-making in autonomous driving. More details about the environment can be seen at https://github. com/eleurent/highway-env.
In this experiment, we focus on the style representation of different drivers in the car-following scenario, in which there is no lane-changing behavior. The car-following scenario in the highwayenv environment is shown in Figure 1(b). In this environment, the initial position and velocity of surrounding vehicles are random. Moreover, the acceleration of the surrounding agents is given by the Intelligent Driver Model (IDM) (Treiber et al., 2000), and the discrete lane change decisions are given by the Minimizing Overall Braking Induced by Lane change (MOBIL) model (Kesting et al., 2007). Some literatures point out that the traditional car following style can be divided into aggressive and conservative. While in this paper, we also introduce a car-following style, that is, the subject vehicle always keeps the minimum driving speed allowed on the highway (20 m/s) for car-following maneuver. It is assumed that the initial lane of the target vehicle is the middle lane and the initial speed is 25m/s. We collect some demonstration data by a policy trained with PPO and reward shaping that can reflect the above different driving styles. Furthermore, the average velocity, the average distance between the ego-vehicle and the leading vehicle and the total driving distance can be used to identify the different driving styles. Table 2 presents the descriptive statistics of car-following behaviors from the expert demonstrations.
Table 2: Descriptive statistics for different driving styles in car-following manuevers
DRIVING STYLE AVERAGE VELOCITY MEAN GAP TOTAL DISTANCE
Aggressive 26.29m/s 35.05m 527.48m Conservative 25.25m/s 65.83m 505.47m Keep speed 20.05m/s 106.58m 403.01m

heading: 4.3.2 EXPERIMENTAL ANALYSIS
text: We use the Style BC method to learn three style embedding, and each discrete index is used to interact with the highway environment to generate a set of specific style trajectory. Results are provided in Figure 8, where the horizontal axis represents the number of time steps and the vertical axis represents the average speed, mean gap and total distance, respectively. The solid blue line represents aggressive driving style, the dotted red line means conservative style and the green point line expresses maintaining the lowest velocity. In Figure 8(a), the velocity of aggressive driving style fluctuates more violently than the other two models. Accordingly, the distance between the following and the leading vehicle is relatively small in aggressive driving style. The following distance of the conservative driving model is stable at 60m, while the driving total distance of the lowest speed driving style is also the shortest. Furthermore, from Figure 8, we can also find that the three driving styles reproduced are also clearly distinguished in various indicators, which shows that the style embedding can guide the generation of different driving strategies. We recorded the driving process of Style BC with a video which can be found at https: //www.bilibili.com/video/BV1Sf4y1F72P?spm_id_from=444.41.0.0. During the interaction between the model and the highway environment, these car-following strategies for different driving style can safely complete the whole journey. While the traditional BC, GAIL and info-GAIL methods will collide with surrounding vehicles in the test stage, so they are not included in the comparative experiment.

heading: 5 CONCLUSION
text: In this paper, we present a method to infer the latent representation of behavior styles automatically while imitating different style policies from expert demonstrations. The novel Style BC method introduces style embedding and proves that it can improve the accuracy of behavior cloning. Our experimental results in some popular benchmark environments show that our methods can reproduce more behavior styles, while an exact style embedding vector can determine a unique style policy. In future work, we seek to explain the performance of different style embedding and find the potential relationship between these style indexes. Besides, extending our method with popular inverse reinforcement learning is also worth researching in the future.

Review summaries:
The paper is about imitation learning, and it particular it considers the task of replicating the style of behaviours. The authors propose a novel framework, called Style Behavioural Cloning that can learn the representation of different behaviour styles from demonstrations by experts. The approach is an extension of behaviour cloning, that tries to derive the behaviour style embedding by optimising the model to predict the action accurately. The paper presents the mathematical formulation of the framework and an experimental analysis of different settings.

The paper proposes a method to learn different behavior styles from expert demonstrations via imitation learning. To achieve this, the method learns a latent style embedding of behavior styles and condition the policy on it. A style policy that takes the style embedding as additional input is learned alongside a target policy without the style embedding. Both policies are trained with behavior cloning (BC). Experiments are performed on simple environments such as Grid-World and Cartpole.

This paper presents Style BC, an extension to standard behavior cloning that uses an additional style embedding to capture different modes in expert demonstrations. The style embeddings are discrete, fixed-size vectors which the policy uses to predict actions from states. Style BC trains two imitative policies: the target policy and the style policy. The target policy directly predicts actions from states; the style policy additionally takes as input the selected style embedding. The style embeddings are trained by back-propagating action prediction gradients. Style BC outperforms the selected baselines (BC, GAIL, InfoGAIL) on both diversity and final performance in the CartPole environment. It is shown to be able to capture different modes from expert demonstrations in the Highway environment.  

This paper focuses on learning different behavior styles from expert demonstrations for imitation learning. More specifically, the contributions are three folds: 1. This paper proposes a novel style behavior cloning approach that learns the various representation of behavior styles from expert demonstrations; 2.  The style embedding which can provide valuable guidance for the generation of style policies is obtained automatically; 3.  The authors empirically demonstrate the effectiveness of the proposed method on a Grid-World environment, cartpole control tasks from OpenAI gym, and a collection of environments for decision-making in autonomous driving named highway-env.