Learning the Representation of Behavior Styles with Imitation Learning

Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions . However , behavior styles such as motion trajectory and driving habit depend largely on the dataset of human maneuvers , and settle down to an average behavior style in most imitation learning algorithms . In this study , we propose a method named style behavior cloning ( Style BC ) , which can not only infer the latent representation of behavior styles automatically , but also imitate different style policies from expert demonstrations . Our method is inspired by the word2vec algorithm and we construct a behavior-style to action mapping which is similar to the word-embedding to context mapping in word2vec . Empirical results on popular benchmark environments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly . Furthermore , compared with various baselines , our policy influenced by its assigned style embedding can better reproduce the expert behavior styles , especially in the complex environments or the number of the behavior styles is large . 1 INTRODUCTION . Deep reinforcement learning ( DRL ) , which combines reinforcement learning and deep neural networks , has achieved rapid development in the last few years . Research institutions as well as industrial companies have applied this technology to many domains , such as games ( Atari games and Go ) ( Silver et al. , 2016 ; Van Hasselt et al. , 2016 ) , robotics control ( Zhang et al. , 2015 ) and autonomous driving ( González et al. , 2016 ) . However , as the DRL algorithms optimize the models according to the reward function , it may not always generate desirable behaviors due to the inferior local convergence or inappropriate reward design , especially in the real world problems where the reward function is difficult to clearly define . Imitation learning ( IL ) methods have the potential to cover this shortage , with which an agent learns to perform a task from expert demonstrations by mimicking a mapping between observations and actions instead of maximizing the accumulated reward ( Osa et al. , 2018 ) . In these methods , the behavior style such as the motion trajectory and operation habit is determined depending on the dataset of expert demonstrations . In practise , the real world problems such as self-driving vehicles ( Munoz et al. , 2009 ) , assistive robots ( Ikemoto et al. , 2012 ) and human-computer interaction ( Ross & Bagnell , 2010 ) may need a large amount of demonstrations and these demonstrations are usually collected from multiple human experts who may have different behavior styles . For the conventional IL algorithms , the imitated behavior tends to converge into an average of the multiple behavior styles in the demonstrations when the imitation model is trained . However , demonstrations from different experts can show significant variability because different human experts typically employ different policies according to their own styles and habits , and learning an average behavior style maybe unacceptable . For instance , in a driving scenario with a leading vehicle , aggressive drivers may make the decision to change lanes . While some drivers are relatively conservative and will maintain a stable following distance with the front vehicle by controlling the velocity , as illustrated in Figure 1 . In these situations , the average behavior reproduced by imitation learning may not express any kind of expert styles , or even worse results in an non-convergent policy ( e.g. , crashing the leading car in the scenario of Figure 1 ) . There are some algorithms considering behavior style in imitation learning . A classical way to learn from demonstration is Dynamic Bayesian Network ( Eppner et al. , 2009 ) , which does not use deep learning . Another way is the constrained methods , such as Probabilistic Principle Component Analysis ( Perico et al. , 2019 ) , which can take into account additional constraints such as speed limits and obstacles . Although these algorithms can reproduce the behavior style from a small number of demonstrations , due to a number of assumptions about the structure of sensory and control signals , such traditional methods are not scalable to more multi-dimensional and multi-modal systems , such as systems involving tactile sensors and raw images . Alternatively , methods based on Generative Adversarial Imitation Learning ( GAIL ) combine supervised and reinforcement learning ( Ho & Ermon , 2016 ) , which produces similar behaviors to the expert demonstrations in a simulation environment . Based on this , Information Maximizing GAIL ( Info-GAIL ) ( Li et al. , 2017 ) , which can embed differences in behavior style in latent space using mutual information , has been used to produce different driver models in an autonomous driving environment . However , InfoGAIL involves sampling a latent code at the beginning of each trial . While when the simulated surrounding vehicles are initialized with the real driving environment information ( such as velocity and position ) , the random sampling of latent codes may not ensure consistency between the policy ’ s subsequent actions and the driver ’ s behaviors . To address this limitation , a Burn-InfoGAIL algorithm ( Kuefler & Kochenderfer , 2017 ) , built on Info-GAIL , draws latent features directly from a learned , inference distribution . In addition , this burn-in policy must take over from the point at which a specific expert demonstration ends . Building upon the above researches , this paper proposes a novel imitation learning framework named Style Behavior Clone ( Style BC ) , which can learn the representation of various behavior styles from expert demonstrations . Not only can the model automatically learn a style embedding that is meaningful to represent the behavior styles , but also accurately reproduces different expert behaviors . Our approach is an extension of behavior cloning ( BC ) ( Bojarski et al. , 2016 ) and different from current style involved imitation learning algorithms which try to infer the style embeddings through inputting the behavior sequences , our method try to solve this problem in a more direct way : the correct embeddings should improve the final performance of BC and thus designing a framework which can adjust the style embeddings and the policy network simultaneously according the final training performance . To be specific , this paper introduces a hypothesis that each policy has a latent feature c , i.e . style embedding , which can determine its behavior style . Therefore , in order to improve the prediction accuracy of the actions belonging to different styles , we need an accurate c to guide the behavior . Our method is inspired by the word2vec ( Mikolov et al. , 2013 ) algorithm which is used to embed the words in natural language processing ( NLP ) . The word2vec algorithm try to compute the word embedding through optimizing the model to predict the contexts accurately , and correspondingly , Style BC tries to compute the behavior style embedding through optimize the model to predict the action accurately . Finally , it is worth noting that our algorithm updates style embedding by making the distinction of behaviors more clear , while Info-GAIL and related deformations disentangle the latent c by inferring from the sequences of state-actions pairs . The main contributions of this paper are summarized as follows : ( i ) we propose a novel style behavior cloning approach that learns the various representation of behavior styles from expert demonstrations ; ( ii ) the style embedding which can provide valuable guidance for the generation of style policies is obtained automatically ; ( iii ) finally , we empirically demonstrate the effectiveness of our research on a Grid-World environment , cartpole control tasks from OpenAI gym , and a collection of environment for decision-making in autonomous driving named highway-env . We show that our model can learn to represent different kinds of behavior styles by exploring the style embedding . 2 PROBLEM FORMULATION . In this section , we start with a brief introduction to the background including Markov Decision Process ( MDP ) and style Markov decision process ( style MDP ) . Then we provide a basic objective function of behavior cloning to learn human behaviors from demonstrations . However , due to the expert demonstrations may collect a variety of behaviors , the simple supervised policy from offline demonstration may not have the ability to reproduce these motion styles . 2.1 BACKGROUND . RL problems are often modeled as Markov Decision Processes ( MDPs ) which are defined as a tuple M = ( S , A , P , R , γ ) . S and A represent states and actions respectively . P is a state transition probability function , which can also be represented as P ( s′|s , a ) . Further R : S × A → R is a reward function measuring the performance of agents and γ is a discount factor for future rewards . Given an MDPM , the goal of the agent is to maximize the expectation of the discounted accumulative reward JR = E [ ∑∞ t=0 γ tr ( st , at , st+1 ) ] automatically by finding an optimal policy π∗ . In this paper , we assume that the behavior policy is a mixture of behavior styles∏ E= { πB1 ( .|s , c1 ) , πB2 ( .|s , c2 ) . . . πBn ( .|s , cn ) } , where c is a style embedding that can determine the behavior style of policy . The objective is to reproduce a policy π ( a|s , c ) as an approximation of πBi , and we use the tuple ( S , A , C , P , R , γ ) to define this process , named style MDP . 2.2 BEHAVIORAL CLONING . BC as a representative imitation learning method seeks the best policy that can minimize the action prediction error in demonstrations . Traditional BC method learns a policy through supervised learning and the only requirements are pairs of input sensory observations associated with expert actions . The objective is to minimize the distribution discrepancy between the expert demonstrations and the policy , which can be formulated as follows : min π D [ ρπ ( ·|s ) || ρE ( ·|s ) ] , ( 1 ) where s represents the environment state and D is the discrepancy measure like KL divergence which is a common objective function in imitation learning . In addition , ρπ and ρE depict the state action distribution generated by policy π and human demonstration E , respectively . Human demonstrations can show significant variability because these demonstrations might be collected from users with different skills and habits . Since the policy is tied up to the demonstration in this objective function , in training the BC tends to converge into an average of various human demonstrations or a fixed behavior style when some behavior style dominates the overall demonstrations . However , sometimes we are more concerned with how to reproduce the variation of human behavior styles . Therefore , inspired by style MDP , we try to improve the prediction accuracy of the model by learning the style embedding automatically , and leverage this encode to explore the variation of behavior styles . 3 PROPOSED METHOD . To learn the style embedding that captures the semantics of human behavior , our main insight is that the representation of styles can be reflected by their motions on the environment . In this paper , we assume that different style episodes in expert demonstrations are generated by a specific style policy πB ( a|s , c ) . And what Style BC tries to do is to find the right behavior style embedding c for each episode in demonstration as well as learn a policy that can output the same action conditioning on the that c and the state as πB ( a|s , c ) does . Figure 2 illustrates the overall framework of our method , which can be divided into two parts : learning the behavior style embedding and learning the imitation policy based on that embedding . The first part is implemented through the backward propagation process in the bottom of Figure 2 . And for the learning of the imitation policy based on the embedding vector , we introduce an additional target policy to learn the representation of the state in the demonstration , which can also make the optimization process of the imitation policy more stable . An outline for the Style BC is shown in Algorithm 1 ..

Summary: This paper presents Style BC, an extension to standard behavior cloning that uses an additional style embedding to capture different modes in expert demonstrations. The style embeddings are discrete, fixed-size vectors which the policy uses to predict actions from states. Style BC trains two imitative policies: the target policy and the style policy. The target policy directly predicts actions from states; the style policy additionally takes as input the selected style embedding. The style embeddings are trained by back-propagating action prediction gradients. Style BC outperforms the selected baselines (BC, GAIL, InfoGAIL) on both diversity and final performance in the CartPole environment. It is shown to be able to capture different modes from expert demonstrations in the Highway environment.