Learning the Representation of Behavior Styles with Imitation Learning

Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions . However , behavior styles such as motion trajectory and driving habit depend largely on the dataset of human maneuvers , and settle down to an average behavior style in most imitation learning algorithms . In this study , we propose a method named style behavior cloning ( Style BC ) , which can not only infer the latent representation of behavior styles automatically , but also imitate different style policies from expert demonstrations . Our method is inspired by the word2vec algorithm and we construct a behavior-style to action mapping which is similar to the word-embedding to context mapping in word2vec . Empirical results on popular benchmark environments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly . Furthermore , compared with various baselines , our policy influenced by its assigned style embedding can better reproduce the expert behavior styles , especially in the complex environments or the number of the behavior styles is large . 1 INTRODUCTION . Deep reinforcement learning ( DRL ) , which combines reinforcement learning and deep neural networks , has achieved rapid development in the last few years . Research institutions as well as industrial companies have applied this technology to many domains , such as games ( Atari games and Go ) ( Silver et al. , 2016 ; Van Hasselt et al. , 2016 ) , robotics control ( Zhang et al. , 2015 ) and autonomous driving ( González et al. , 2016 ) . However , as the DRL algorithms optimize the models according to the reward function , it may not always generate desirable behaviors due to the inferior local convergence or inappropriate reward design , especially in the real world problems where the reward function is difficult to clearly define . Imitation learning ( IL ) methods have the potential to cover this shortage , with which an agent learns to perform a task from expert demonstrations by mimicking a mapping between observations and actions instead of maximizing the accumulated reward ( Osa et al. , 2018 ) . In these methods , the behavior style such as the motion trajectory and operation habit is determined depending on the dataset of expert demonstrations . In practise , the real world problems such as self-driving vehicles ( Munoz et al. , 2009 ) , assistive robots ( Ikemoto et al. , 2012 ) and human-computer interaction ( Ross & Bagnell , 2010 ) may need a large amount of demonstrations and these demonstrations are usually collected from multiple human experts who may have different behavior styles . For the conventional IL algorithms , the imitated behavior tends to converge into an average of the multiple behavior styles in the demonstrations when the imitation model is trained . However , demonstrations from different experts can show significant variability because different human experts typically employ different policies according to their own styles and habits , and learning an average behavior style maybe unacceptable . For instance , in a driving scenario with a leading vehicle , aggressive drivers may make the decision to change lanes . While some drivers are relatively conservative and will maintain a stable following distance with the front vehicle by controlling the velocity , as illustrated in Figure 1 . In these situations , the average behavior reproduced by imitation learning may not express any kind of expert styles , or even worse results in an non-convergent policy ( e.g. , crashing the leading car in the scenario of Figure 1 ) . There are some algorithms considering behavior style in imitation learning . A classical way to learn from demonstration is Dynamic Bayesian Network ( Eppner et al. , 2009 ) , which does not use deep learning . Another way is the constrained methods , such as Probabilistic Principle Component Analysis ( Perico et al. , 2019 ) , which can take into account additional constraints such as speed limits and obstacles . Although these algorithms can reproduce the behavior style from a small number of demonstrations , due to a number of assumptions about the structure of sensory and control signals , such traditional methods are not scalable to more multi-dimensional and multi-modal systems , such as systems involving tactile sensors and raw images . Alternatively , methods based on Generative Adversarial Imitation Learning ( GAIL ) combine supervised and reinforcement learning ( Ho & Ermon , 2016 ) , which produces similar behaviors to the expert demonstrations in a simulation environment . Based on this , Information Maximizing GAIL ( Info-GAIL ) ( Li et al. , 2017 ) , which can embed differences in behavior style in latent space using mutual information , has been used to produce different driver models in an autonomous driving environment . However , InfoGAIL involves sampling a latent code at the beginning of each trial . While when the simulated surrounding vehicles are initialized with the real driving environment information ( such as velocity and position ) , the random sampling of latent codes may not ensure consistency between the policy ’ s subsequent actions and the driver ’ s behaviors . To address this limitation , a Burn-InfoGAIL algorithm ( Kuefler & Kochenderfer , 2017 ) , built on Info-GAIL , draws latent features directly from a learned , inference distribution . In addition , this burn-in policy must take over from the point at which a specific expert demonstration ends . Building upon the above researches , this paper proposes a novel imitation learning framework named Style Behavior Clone ( Style BC ) , which can learn the representation of various behavior styles from expert demonstrations . Not only can the model automatically learn a style embedding that is meaningful to represent the behavior styles , but also accurately reproduces different expert behaviors . Our approach is an extension of behavior cloning ( BC ) ( Bojarski et al. , 2016 ) and different from current style involved imitation learning algorithms which try to infer the style embeddings through inputting the behavior sequences , our method try to solve this problem in a more direct way : the correct embeddings should improve the final performance of BC and thus designing a framework which can adjust the style embeddings and the policy network simultaneously according the final training performance . To be specific , this paper introduces a hypothesis that each policy has a latent feature c , i.e . style embedding , which can determine its behavior style . Therefore , in order to improve the prediction accuracy of the actions belonging to different styles , we need an accurate c to guide the behavior . Our method is inspired by the word2vec ( Mikolov et al. , 2013 ) algorithm which is used to embed the words in natural language processing ( NLP ) . The word2vec algorithm try to compute the word embedding through optimizing the model to predict the contexts accurately , and correspondingly , Style BC tries to compute the behavior style embedding through optimize the model to predict the action accurately . Finally , it is worth noting that our algorithm updates style embedding by making the distinction of behaviors more clear , while Info-GAIL and related deformations disentangle the latent c by inferring from the sequences of state-actions pairs . The main contributions of this paper are summarized as follows : ( i ) we propose a novel style behavior cloning approach that learns the various representation of behavior styles from expert demonstrations ; ( ii ) the style embedding which can provide valuable guidance for the generation of style policies is obtained automatically ; ( iii ) finally , we empirically demonstrate the effectiveness of our research on a Grid-World environment , cartpole control tasks from OpenAI gym , and a collection of environment for decision-making in autonomous driving named highway-env . We show that our model can learn to represent different kinds of behavior styles by exploring the style embedding . 2 PROBLEM FORMULATION . In this section , we start with a brief introduction to the background including Markov Decision Process ( MDP ) and style Markov decision process ( style MDP ) . Then we provide a basic objective function of behavior cloning to learn human behaviors from demonstrations . However , due to the expert demonstrations may collect a variety of behaviors , the simple supervised policy from offline demonstration may not have the ability to reproduce these motion styles . 2.1 BACKGROUND . RL problems are often modeled as Markov Decision Processes ( MDPs ) which are defined as a tuple M = ( S , A , P , R , γ ) . S and A represent states and actions respectively . P is a state transition probability function , which can also be represented as P ( s′|s , a ) . Further R : S × A → R is a reward function measuring the performance of agents and γ is a discount factor for future rewards . Given an MDPM , the goal of the agent is to maximize the expectation of the discounted accumulative reward JR = E [ ∑∞ t=0 γ tr ( st , at , st+1 ) ] automatically by finding an optimal policy π∗ . In this paper , we assume that the behavior policy is a mixture of behavior styles∏ E= { πB1 ( .|s , c1 ) , πB2 ( .|s , c2 ) . . . πBn ( .|s , cn ) } , where c is a style embedding that can determine the behavior style of policy . The objective is to reproduce a policy π ( a|s , c ) as an approximation of πBi , and we use the tuple ( S , A , C , P , R , γ ) to define this process , named style MDP . 2.2 BEHAVIORAL CLONING . BC as a representative imitation learning method seeks the best policy that can minimize the action prediction error in demonstrations . Traditional BC method learns a policy through supervised learning and the only requirements are pairs of input sensory observations associated with expert actions . The objective is to minimize the distribution discrepancy between the expert demonstrations and the policy , which can be formulated as follows : min π D [ ρπ ( ·|s ) || ρE ( ·|s ) ] , ( 1 ) where s represents the environment state and D is the discrepancy measure like KL divergence which is a common objective function in imitation learning . In addition , ρπ and ρE depict the state action distribution generated by policy π and human demonstration E , respectively . Human demonstrations can show significant variability because these demonstrations might be collected from users with different skills and habits . Since the policy is tied up to the demonstration in this objective function , in training the BC tends to converge into an average of various human demonstrations or a fixed behavior style when some behavior style dominates the overall demonstrations . However , sometimes we are more concerned with how to reproduce the variation of human behavior styles . Therefore , inspired by style MDP , we try to improve the prediction accuracy of the model by learning the style embedding automatically , and leverage this encode to explore the variation of behavior styles . 3 PROPOSED METHOD . To learn the style embedding that captures the semantics of human behavior , our main insight is that the representation of styles can be reflected by their motions on the environment . In this paper , we assume that different style episodes in expert demonstrations are generated by a specific style policy πB ( a|s , c ) . And what Style BC tries to do is to find the right behavior style embedding c for each episode in demonstration as well as learn a policy that can output the same action conditioning on the that c and the state as πB ( a|s , c ) does . Figure 2 illustrates the overall framework of our method , which can be divided into two parts : learning the behavior style embedding and learning the imitation policy based on that embedding . The first part is implemented through the backward propagation process in the bottom of Figure 2 . And for the learning of the imitation policy based on the embedding vector , we introduce an additional target policy to learn the representation of the state in the demonstration , which can also make the optimization process of the imitation policy more stable . An outline for the Style BC is shown in Algorithm 1 ..

Summary: The paper proposes a method to learn different behavior styles from expert demonstrations via imitation learning. To achieve this, the method learns a latent style embedding of behavior styles and condition the policy on it. A style policy that takes the style embedding as additional input is learned alongside a target policy without the style embedding. Both policies are trained with behavior cloning (BC). Experiments are performed on simple environments such as Grid-World and Cartpole.