Learning the Representation of Behavior Styles with Imitation Learning
Authors: Xiao Liu
Meng Wang
Zhaorong Wang
Yingfeng Chen
Yujing Hu
Changjie Fan
Chongjie Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Imitation Learning
Behavior Style

Abstract: Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions. However, behavior styles such as motion trajectory and driving habit depend largely on the dataset of human maneuvers, and settle down to an average behavior style in most imitation learning algorithms. In this study, we propose a method named style behavior cloning (Style BC), which can not only infer the latent representation of behavior styles automatically, but also imitate different style policies from expert demonstrations. Our method is inspired by the word2vec algorithm and we construct a behavior-style to action mapping which is similar to the word-embedding to context mapping in word2vec. Empirical results on popular benchmark environments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly. Furthermore, compared with various baselines, our policy influenced by its assigned style embedding can better reproduce the expert behavior styles, especially in the complex environments or the number of the behavior styles is large.

Under review as a conference paper at ICLR 2022
LEARNING THE REPRESENTATION OF BEHAVIOR STYLES WITH IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation learning is one of the methods for reproducing expert demonstrations adaptively by learning a mapping between observations and actions. However, behavior styles such motion trajectory and driving habit depend largely the dataset of human maneuvers, and settle down to an average behavior style most imitation learning algorithms. In this study, we propose method named style behavior cloning (Style BC), which can not only infer the latent representation behavior styles automatically, but also imitate different style policies from expert demonstrations. Our method is inspired by the word2vec algorithm and we con- struct behavior-style to context mapping in word2vec. Empirical results on popular benchmark envi- ronments show that Style BC outperforms standard behavior cloning in prediction accuracy and expected reward significantly. Furthermore, compared with various baselines, our policy influenced by its assigned style embedding can better repro- duce the expert behavior styles, especially in the complex environments the number of the behavior styles is large.
action mapping which is similar to the word-embedding
INTRODUCTION
Deep reinforcement learning (DRL), which combines reinforcement learning and deep neural net- works, has achieved rapid development in the last few years. Research institutions as well as indus- trial companies have applied this technology to many domains, such as games (Atari games and Go) (Silver et al. 2016 Van Hasselt al. 2016). robotics control (Zhang et al., 2015) and autonomous driving (González et al. (016). However, as the DRL algorithms optimize the models according to the reward function, it may not always generate desirable behaviors due the inferior local con- vergence inappropriate reward design, especially in the real world problems where the reward function is difficult to clearly define.
Imitation learning (II.) methods have the potential to cover this shortage, with which agent learns to perform a task from expert demonstrations by mimicking a mapping between observations and actions instead of maximizing the accumulated reward (Osa et al. 2018). In these methods, the behavior style such as the motion trajectory and operation habit is determined depending on the dataset of expert demonstrations. In practise, the real world problems such as self-driving vehicles (Munoz et al. 2009), assistive robots (Ikemoto et al.) (002)) and human-compute interaction (Ross) & | 2010) may need large amount of demonstrations and these demonstrations collected from multiple human experts who may have different behavior styles. For the conventional IL algorithms, the imitated behavior tends to converge into an average of the multiple behavior styles in the demonstrations when the imitation model is trained. However, demonstrations from different experts can show significant variability because different human experts typically employ different policies according to their own styles and habits, and learning an average behavior style maybe unacceptable. For instance, a driving scenario with a leading vehicle, aggressive drivers may make the decision to change lanes. While some drivers are relatively conservative and will maintain a stable following distance with the front vehicle by controlling the velocity, as illustrated in Figure I In these situations, the average behavior reproduced by imitation learning may not express any kind expert styles, or even worse results in an conceenverg policy (e.g., crashing the leading in the scenario of Figure T).
usually

---

Under review as a conference paper at ICLR 2022
(a)
(b)
Figure 1: Driving behaviors of human: (a) Lane-changing; (b) Car- followil..
There are some algorithms considering behavior style in imitation learning. A classical way to learn from demonstration is Dynamic Bayesian Network (Eppner et al., 2009), which does not use deep learning. Another way is the constrained methods, such as Probabilistic Principle Component Anal- ysis (Perico et al. 2019), which can take into account additional constraints such as speed limits and obstacles. Although these algorithms can reproduce the behavior style from a small number of demonstrations, due number of assumptions about the structure of sensory and control sig- nals, such traditional methods are not scalable to more multi-dimensional and multi-modal systems, such as systems involving tactile sensors and raw images. Alternatively, methods based on Genera- tive Adversarial Imitation Learning (GAIL) combine supervised and reinforcement learning (Ho & Ermon 2016), which produces similar behaviors to the expert demonstrations in a simulation envi- ronment. Based on this, Information Maximizing GAIL (Info-GAIL) (Li et al. 2017), which can embed differences in behavior style in latent space using mutual information, has been used to pro- duce different driver models in an autonomous driving environment. However, InfoGAIL involves sampling a latent code at the beginning of each trial. While when the simulated surrounding ve- hicles are initialized with the real driving environment information (such as velocity and position), the random sampling of latent codes may not ensure consistency between the policy's subsequent actions and the driver's behaviors. To address this limitation, Burn-InfoGAIL algorithm (Kuefler & Kochenderfer 2017), built on Info-GAIL, draws latent features directly from a learned, inference distribution, In addition, this burn-in policy must take over from the point at which specific expert demonstration ends.
Building upon the above researches, this paper proposes a novel imitation learning framework named Style Behavior Clone (Style BC), which can learn the representation of various behavior styles from expert demonstrations. Not only can the model automatically learn a style embedding that mean- ingful to represent the behavior styles, but also accurately reproduces different expert behaviors. Our approach an extension of behavior cloning (BC) (Bojarski et al. 2016 and different from current style involved imitation learning algorithms which try to infer the style embeddings through inputting the behavior sequences, our method try to solve this problem in a more direct way: the correct embeddings should improve the final performance of BC and thus designing framework which can adjust the style embeddings and the policy network simultaneously according the final training performance.
To be specific, this paper introduces hypothesis that each policy has a latent feature c, i.e. style embedding, which can determine its behavior style. Therefore, in order improve the prediction accuracy of the actions belonging to different styles, we need an accurate to guide the behavior. Our method is inspired by the word2vec (Mikolov al. 2013) algorithm which used embed the words in natural language processing (NLP). The word2vec algorithm try to compute the word embedding through optimizing the model to predict the contexts accurately, and correspondingly, Style BC tries to compute the behavior style embedding through optimize the model to predict the action accurately. Finally,
worth noting that our algorithm updates style embedding by making the distinction of behaviors more clear, while Info-GAll and related deformations disentangle the latent e by inferring from the sequences of state-actions pairs.
The main contributions of this paper are summarized as follows: (i) propose novel style behav- ior cloning approach that learns the various representation of behavior styles from expert demon- strations; (ii) the style embedding which can provide valuable guidance for the generation of style policies is obtained automatically; (iii) finally, we empirically demonstrate the effectiveness of our research Grid- World environment, cartpole control tasks from OpenAl gym, and a collection
2

---

more concerned with how reproduce the variation of human Under review as a conference paper at ICLR 2022
of environment for decision decision-making making in autonomous driving named highway-env. We show that our model can learn to represent different kinds of behavior styles by exploring the style embedding.
PROBLEM FORMULATION
In this section, we start with a brief introduction to the background including Markov Decision Process (MDP) and style Markov decision process (style MDP). Then provide basic objective function behavior cloning to learn human behaviors from demonstrations. However, due to the expert demonstrations may collect variety of behaviors, the simple supervised policy from offline demonstration may not have the ability to reproduce these motion styles.
2.1 BACKGROUND
RL problems are often modeled as Markov Decision Processes (MDPs) which are defined tuple M = (, R.,1 7). S and A represent states and actions respectively. P is state transition probability function, which can also be represented as P(s'ls, |s, a). Further R
A IR is reward function measuring the performance of agents and is discount factor for future rewards. Given an MDP M, the goal of the agent is to maximize the expectation the discounted accumulative reward JR E 12t= rr((st, dt, 8t41)] automatically by finding an optimal policy .².
In this paper,
assume that the behavior policy is mixture of behavior styles I {TB, (1),,c) TB, (2), ...... TB, (-|s, cn) }. where e is a style embedding that can deter- mine the behavior style of policy. The objective is to reproduce policy л (a]ss, c) approxima- tion of TB. and we use the tuple (S, C, R, 7) to define this process, named style MDP.
2.2 BEHAVIORAL CLONING
BC as a representative imitation learning method seeks the best policy that can minimize the action prediction етrоr in demonstrations. Traditional BC method learns policy through supervised learn- ing and the only requirements are pairs of input sensory observations associated with expert actions. The objective is to minimize the distribution discrepancy between the expert demonstrations and the policy, which can be formulated as follows:
min [px (1
where represents the environment state and ID is the discrepancy measure like KL divergence which is a common objective function in imitation learming. In addition, and depict the state action distribution generated by policy и and human demonstration F., respectively.
Human demonstrations can show significant variability because these demonstrations might be col- lected from users with different skills and habits. Since the policy tied up to the demonstration in this objective function, in training the BC tends to converge into average of various human demonstrations or fixed behavior style when some behavior style dominates the overall demon- strations. However, sometimes behavior styles.
Therefore, inspired by style MDP, we try to improve the prediction accuracy of the model by learning the style embedding automatically, and leverage this encode to explore the variation of behavior styles.
PROPOSED METHOD
To learn the style embedding that captures the semantics of human behavior, main insight is that the representation of styles can reflected by their motions on the environment. In this paper, assume that different style episodes in expert demonstrations are generated by specific style policy n (a) s, c). And what Style BC tries to do is to find the right behavior style embedding e for each episode in demonstration well learn policy that output the same action conditioning
3

---

Under review as a conference paper at ICLR 2022
Collacted
Training
Achattor
action
target pallet
shar layers
palloy
Accion
backward
baclward propegation
copy
Expart
Action
different
sky policy
shared layers
polley leyers
Action
Lusss
backswerd propagation
cmbedding
backward propagation
Figure 2: Overview of the framework of our proposed method.
on the that and the state as Ty (als, does. Figure illustrates the overall framework of our method, which can be divided into two parts: learning the behavior style embedding and learning imitation policy based on that embedding. The first part implemented through the backward propagation process in the bottom of Figure 2 And for the learning of the imitation policy based on the embedding vector, we introduce an additional target policy to learn the representation of the state in the demonstration, which can also make the optimization process of the imitation policy more stable. An outline for the Style BC shown in Algorithm
3.1 THE LEARNING OF THE BEHAVIOR STYLE EMBEDDING
Firstly, in order to initialize the behavior style vectors C, we have to determine the number of the behavior styles before training. We can determine in two ways: 1) if the number of the episodes in the demonstrations is not large, we set N to be the number of the episodes in the demonstrations; 2) otherwise if the number of the episodes in the demonstration large, we can human domain knowledge, and will introduce
determine proper value of N according method to select a right behavior embedding for each episode in the demonstrations Section 3.3. We take N hyper-parameter and this is the only domain knowledge related parameter in framework.
After selecting the style vector for each episode, have optimize this vector help the model predicting the actions in this episodes more accurately. We leverage the widely used supervised
Algorithm 1:
Style BC
Input: N:
the number of the predetermined styles, E: the expert demonstration set, Otarger and the network weights of the shared layers and the policy layers in target network, 0 Ostyle and Wstyle: the network weights of the shared layers and the policy
Wtarget:
layers in style network
Initialize Otargets Whorget, Ostyle and Wstyles random initialize N behavior style vectors.
Select an expert episode e from E, and determine the style vector of this episode according to
Eq. 6
Update the target network on according to Eq.
Update the style vector and the policy layers Wstyle of style policy according to Eq. and Eq.
3
Update the shared layers Osryle style policy according to Eq.
4

---

Under review as a conference paper at ICLR 2022
learning process for this purpose. As illustrated in Figure the network named style policy is responsible for this process and it can be divided into two parts: the shared layers parameterized with style and the policy layers parameterized with Wstyle We introduce the imitation learning loss to update e and the optimization can be formulated as:
min D (((stticcWsssae (-]s,c) || An (-|s,c)]
(2)
where TB (-|s,c) c) is the action distribution calculated from the episode with behavior style vector C.
3.2 THE LEARNING OF THE IMITATION POLICY
Next, we will introduce the process of optimizing the imitation policy, which is denoted style pol- icy in Figure 2 As mentioned above, the two network modules of it named shared layers and policy layers are parameterized with Sstyle: and Wstyle respectively. For Ostyles the optimization process is similar to the process in optimizing the behavior style embedding which can be formulated as:
min TD (((atstte
c) || TB (-18,c)],
Matgle
This optimization is the same as in the common behavior clone algorithm and is intuitive to introduce it here a target policy to update it, the target policy is a common behavior clone network with the same network construction as the style policy, except for that its input includes the state only and will not contain the style embedding. The shared layers and the policy layers of it are paremeterized with Orarget and Wtarget respectively. In training, the parameters of the shared layers in style policy are copied from the ones in target policy. We use the target policy here because we take the outputs of the shared layers of the target network as a high level representation of the state and copy it to the style policy can make the optimization process of the style policy faster and more stable. The optimization process of Ostyles Orarget and Witargat can be formatted as:
want the style policy to mimic the expert policy. For dstyles introduce
min IB [ ((0errgcc (eaaget
(+|s) B (-1s)]
(4)
Ostyle k Ostyle (1 k) Otarget
(5)
where k is the policy network parameterized by ftarget and Wtarget. Note that the input of the target policy network Eq. i is only the state s, which different from the input (including both the state s and the behavior style vector c) of the style policy network in Eq. 3
used in the soft update of Ostyle. and (e is the target
3.3 SELECTING THE BEHAVIOR STYLE VECTOR FOR EACH EPISODE
In this section, we will explain the method for selecting the behavior style vector when the number of the episodes is large and thus we have to predetermined the number of the behavior styles before training. After selecting an episode from the expert demonstrations, we will compute the prediction accuracy of the actions in this demonstrated episode with the style policy based on each behavior style vector in C, and select the one with the highest prediction accuracy as the behavior style vector of this episode. To be specific, the behavior style vector ca for episode is determined through Eq. G where Pate Pa (tte (elc) is the prediction accuracy of style policy on the data in episode taking the behavior style vector as input.
arg max Pa CEC
(O w atyle) (elc)
(6)
4 EXPERIMENTS
In this section, we aim at investigating (i) whether Style BC can reproduce the different behavior styles from expert demonstrations, and (ii) whether style embedding can guide and determine the generation of specific style policy. To evaluate our approach, we conduct extensive experiments on several widely used environments: navigation task in GrrldWorld, one classic control environ- ment in OpenAl Gym (Kumar 2020), and finally, a minimalist environment for decision decision-making in autonomous driving named highway-env (Leurent 2018).
5

---

Under review as a conference paper at ICLR 2022
wonter
Agent
20
Target
Figure 3: The human demonstrations the Grid-World environment.
ACC
Reward
Vm
0.90
-100
4
0.55
200
0.80
0.75
-000
30000
100000
15/0000
200000
200000
30009
.00000
1500000 step
200000
2500000
deep
(a)
(b)
Figure 4: Experiment results Style BC and BC in the Grid-world environment. (a) acc and (b) reward.
4.1 GRID-WORLD
4.1.1 EXPERIMENTAL SETUP
We use 15$15 Grid- World environment shown in Figure the agent (yellow point) starts from the bottom left corner and is required to arrive at any of the five goal grid (green squares) placed at the right sideline (+20 reward). Besides, there is wall (black squares) in the vertical direction and the agent can only pass through it from the five doors. We collected trajectories passing through different doors and reaching different five goals, shown in Figure 3 Based this, conduct a set experiments and intend to verify the following issues: (i) Whether the accuracy of behavior cloning be improved after introducing style embedding? (ii) Whether the proposed Style BC method can reproduce different style policies? (iii) there an exact style embedding vector that can determine unique style policy?
For the first issue, by comparing Style BC and traditional BC method the above environment, we observe the differences in accuracy and reward. On this foundation, we introduce two popular imi-

---

Under review as a conference paper at ICLR 2022
tation learning algorithms, Gail and info and count the number of behavior styles displayed by these four algorithms. As for the last issue, several explicit style embedding is given to learn whether different styles of paths are produced.
4.1.2 EXPERIMENTAL ANALYSIS
First of all, from the results in Figure we can find that the accuracy of Style BC is much higher than traditional BC, which proves that the introduction of style embedding will improve the pre- diction accuracy of behavior cloning. Moreover, the advantage of adopting style embedding is also reflected in reward. In the later stage of training, the reward of Style BC tends to be stable. On the contrary, the traditional BC's reward fluctuates greatly because this method sensitive to predic- tion errors. Figure - shows the number of behavior styles displayed by the four algorithmss The demonstration path is a collection of trajectories that pass through five gates and reach different goals, as shown in Figure 3 Therefore, we consider that there are 25 motion strategies in this ex- pert demonstration. Style BC method proposed in this paper reproduces 22 different motion paths, which is the most among the four comparison algorithms: Gail and BC finally converge to only one trajectory, which reflects that the traditional imitation learning algorithm may not be able to show variety of behavior styles; Info-GAIL represents four different motion trajectories, while that some different latent factors correspond to the same behavior style, resulting in only four style paths. In particular, although our method does not completely reproduce the 25 style trajectories, the number of styles reflected is much higher than the four in info-GAlL. Finally, we randomly select three vectors from the style embedding dataset generated by automatic learning the test style. By interacting with the Grid- world environment respectively. three different motion trajectories can be obtained, as shown in Figure Therefore, under the determined style embedding, only one style of behavior policy can be generated.
find
4.2
CARTPOLE
4.2. 1
EXPERIMENT SETUP
This set of experiments are based on the classic control tasks named CartPole-vl in OpenAl Gym. In this research, we assume that keeping the pendulum upright and finally staying at different specific positions (0, +0.5, 1.. distance to center in CartPole) represents various behavior styles, shown in Figure 7 The expert demonstrations collect the trajectories where the end point stays at these five points. In the following experiments, Style BC is trained with the demonstrations and is hoped to reproduce different behavior styles. Similarly, we also introduce BC, GAIL and info-GAIL as the comparison algorithms.
4.2.2 EXPERIMENTAL ANALYSIS
The results of CartPole-vl I are demonstrated in Table [ It's obvious that Style BC and info-GAIL are much higher in style diversity than the other two models. BC learns the behavior style with the end point near -1.5 (-1.46), while algorithm GAIL generates a path with the end stable point near 0 (-0.02). Both behavior styles are from expert demonstrations. Furthermore, we can see that the
XWI
Number of behavior styles
0
STYLE MEER
Figure 5: The number of behavior styles.
Figure 6: Trajectories of the executed policies.
7

---

Under review as a conference paper at ICLR 2022
Figure 7: The Cartpole environment.
trajectory end point shown by Style BC stable different positons, of which four positions stay in our expected position range. This shows that the Style BC model can replicate the four style representations shown by expert demonstrations. generated by Style BC is stable at 0.25. And we believe that the stable point does not fall in the desirable position (0.5) due the deviation of style embedding. the end of the trajectories it produces also falls in five different positions, these end stable points are inconsistent with the expected position. This does not meet the requirements of reproducing different behavior styles.
particular, the end point of one of the tracks
for Info-GAIL model, although
4.3 HIGHWAY-RNY
4.3.1 EXPERIMENT SETUP
To demonstrate the effectiveness of Style BC in complex driving environment, the last experi- ment is carried out Highway-env, which is minimalist environment for decision-making in autonomous driving. More details about the environment can be seen at hhttps: httts:://ithhb.
com//eleuurent/hiiiiwyyyenn
In this experiment, we focus on the style representation of different drivers in the car-following scenario, in which there is lane-changing behavior. The car-following scenario in the highway- env environment is shown in Figure [bb] In this environment, the initial position and velocity of surrounding vehicles are random. Morcover, the acceleration of the surrounding agents is given by the Intelligent Driver Model (IDM) (Treiber et al.) 2000). and the discrete lane change decisions are given by the Minimizing Overall Braking Induced by Lane change (MOBIL) model (Kesting et al., (007). Some literatures point out that the traditional car following style can be divided into aggressive and conservative. While this paper, we also introduce car-following style, that is, the subject vehicle always keeps the minimum driving speed allowed on the highway (20 m/ s) for car-following maneuver. It assumed that the initial lane of the target vehicle the middle lane and the initial speed is 25m/s. We collect some demonstration data by policy trained with PPO and reward shaping that can reflect the above different driving styles. Furthermore, the average velocity, the average distance between the ego-vehicle and the leading vehicle and the total driving distance can be used to identify the different driving styles. Table 2 presents the descriptive statistics of car-following behaviors from the expert demonstrations.
Table 1: Experiment results on Cartpole
ALGORITHM STYLE NUM
FINAL POS
BC Style BC GAIL info-GAIL
[-1.46]
[0.48: 1.46; -1.46; 0.25; -0.08]
[-0.02]
[-2.41; -0.74; -1.38; -0.11; -0.73]
8

---

Under review as a conference paper at ICLR 2022
Table 2: Descriptive statistics for different driving styles in car-following manuevers
DRIVING STYLE AVERAGE VELOCITY
MEAN GAP
TOTAL DISTANCE
Aggressive Conservative Keep seepsppeed
26. 26.29m/s
35.05m
527. 48m
25. 25.25m/
65.83m
505.47m
20.05m/
106.58m
403.01m
srucee server lestrrg
Writhn dirsee
(b)
Figure 8: Experiment results car-following scenario. (a) average speed and (b) mean gap (c) total distance.
4.3.2 EXPERIMENTAL ANALYSIS
We use the Style BC method to learn three style embedding, and each discrete index used to interact with the highway environment to generate set of specific style trajectory. Results are provided in Figure 8. where the horizontal axis represents the number of time steps and the ver- tical axis represents the average speed, mean gap and total distance, respectively. The solid blue line represents aggressive driving style, the dotted red line means conservative style and the green point line expresses maintaining the lowest velocity. In Figure B(a). the velocity of aggressive driving style fluctuates more violently than the other two models. Accordingly, the distance be- tween the following and the leading vehicle is relatively small in aggressive driving style. The following distance of the conservative driving model is stable at 60m, while the driving total dis- tance of the lowest speed driving style is also the shortest. Furthermore, from Figure g. we various indica-
also find that the three driving styles reproduced are also clearly distinguished tors, which shows that the style embedding can guide the generation of different driving strate- gies. We recorded the driving process of Style BC with a video which can be found at https: 7/www. wwwiibiii..cmm com/ v deo BBt/B///S11112///
id fro==444 41 0. During the interaction between the model and the highway environment, these car-following strategies for different driving style can safely complete the whole journey. While the traditional BC, GAIL and info-GAIL. methods will collide with surrounding vehicles in the test stage, so they are not included in the comparative experiment.
CONCLUSION
In this paper, we present method infer the latent representation of behavior styles automatically while imitating different style policies from expert demonstrations. The novel Style BC method introduces style embedding and proves that it can improve the accuracy of behavior cloning. Our experimental results in some popular benchmark environments show that our methods can reproduce more behavior styles, while an exact style embedding vector can determine a unique style policy. In future work, seek to explain the performance of different style embedding and find the poten- tial relationship between these style indexes. Besides, extending our method with popular inverse reinforcement learning is also worth researching in the future.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mariusz. Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End end learning for self-driving cars. arXiv preprint WwXivv1604.033111 2016.
Clemens Eppner, Jurgen Sturm, Maren Bennewitz, Cyrill Stachniss, and Wolfram Burgard. Imitation learning with generalized task descriptions. In 2009 IEEE International Conference on Robotics and Automation, pp. 99744. IEEE, 2009.
David Sierra González, Jilles Steeve Dibangoye, and Christian Laugier. High-speed highway scene prediction based on driver models learned from demonstrations. In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 149-155, 2016. doi: 10.1109/ITSC. A016.7795546.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565-45777 2016.
Shuhei Ikemoto, Heni Ben Amor, Takashi Minato, Bemhard Jung, and Hiroshi Ishiguro. Physical human-robot interaction: Mutual learning and adaptation. IEEE robotics & automation magazine, 19(4):24-35, 2012.
Arne Kesting, Martin Treiber, and Dirk Helbing. General lane-changing model mobil for car- following models. Transportation Research Record, 1999(1):86-94) 94, 2007.
Alex Kuefler and Mykel J Kochenderfer. Burn-in demonstrations for multi-modal imitation learning- arXiv preprint arXiv: 1710.5009, 2017.
Swagat Kumar. Balancing a cartpole system with reinforcement learning-s a tutorial. arXiv preprint 0434:22006.04938 2020.
Gooured An environment for autonomous driving decision-making. https:/// Com/e wew.eeeee//gggwwayaeaeee
2018.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from vi- sual demonstrations. Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 3815-3825, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space, 2013.
Jorge Munoz, German Gutierrez, and Araceli Sanchis. Controller for tores created by imitation. In 2009 IEEE Symposium on Computational Intelligence and Games, pp. 271-278, 2009, doi: 10. 200.0///C......5886666
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. arXiv preprint 1tXiv:1811.067111 2018.
Cristian Alejandro Vergara Perico, Joris De Schutter, and Erwin Aertbeliën. Combining imitation learning with constraint-basel task specification and control. IEEE Robotics and Automation 2019.
Letters,
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning, In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings, 2010.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical review E, 62(2))1805, 2000.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
10

---

Under review as a conference paper at ICLR 2022
Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Uperoft, and Peter Corke. Towards vision- based deep reinforcement learning for robotic motion control, arXiv preprint arXiv:1511.0377 2015.


Reviews:
[{"review_id": "E9-na3r1Lhs", "reviewer": "Reviewer_5Mw5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is generally well written and easy to follow. The motivation of learning multiple different behavior styles is clearly stated with sufficient background information. The proposed method is validated based on multiple benchmark environments and obtains supportive experimental results.\n\nWeaknesses:\n1. In the proposed method, the number of behavior styles is a hyperparameter and the behavior styles need to be pre-defined explicitly based on human prior knowledge. However, in many tasks/scenarios, the behavior styles are hard to define and may even be impossible to be separated into distinct classes. It would be better to discuss the potential generalization to such tasks or the limitations of the proposed method.\n2. In the grid-world environment, what is the value of N? Is it the same as the true number of behavior styles (i.e., 25)? In the testing policies, why do all the policies make turning immediately after passing the door, is it because there is no other turning behavior at other columns in the expert demonstrations?\n3. Following up on the last question, is the proposed method able to explore/learn the feasible behavior styles not present in the expert demonstrations?\n4. The current video demo has no explanation, which is hard to understand the differences between the three behavior styles. It would be better to add some descriptions in the video.", "summary_of_the_paper": "This paper focuses on learning different behavior styles from expert demonstrations for imitation learning. More specifically, the contributions are three folds:\n1. This paper proposes a novel style behavior cloning approach that learns the various representation of behavior styles from expert demonstrations;\n2.  The style embedding which can provide valuable guidance for the generation of style policies is obtained automatically;\n3.  The authors empirically demonstrate the effectiveness of the proposed method on a Grid-World environment, cartpole control tasks from OpenAI gym, and a collection of environments for decision-making in autonomous driving named highway-env.", "summary_of_the_review": "Overall, the idea of learning different behavior styles from expert demonstrations is interesting. However, I would recommend the authors to address my questions in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BRCuMb2M_hX", "reviewer": "Reviewer_5JcU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n\n- Learning different behavior styles could be important for imitation learning to resolve ambiguity when the agent can choose different behaviors in the same state.\n\n**Weakness:**\n\n- The paper is not well written, and there are numerous grammar errors and typos (see detailed comments). The technical section also lacks motivation for each design, making it hard to follow.\n- The idea of the proposed approach also has little novelty. [1] already studied how to learn a latent representation with VAE and use it for behavior cloning (BC) and GAIL. There is also no motivation about why the proposed idea might be better compared to VAE-based approaches for behavior style learning such as [1].\n- The experiments are only performed using very simple environments such as Grid-World, Cart-Pole, and Highway-Env.  In these environments, the method is only compared with all baselines on Cart-Pole. In contrast, similar work [1] has performed experiments with more difficult tasks such as human motion imitation with humanoids, which has high dimensional state and action spaces. Experiments on more and harder MuJoCo tasks as well as comparisons with all baselines need to be provided to demonstrate the method's effectiveness.\n- There are no ablation studies justifying the design of the method, including the use of the target policy, shared layers, and soft update in Eq (5).\n- The method requires selecting the number of behavior styles beforehand, which could be limiting when the number doesn't correspond to the actual number of styles. This again raises the question of why not just use VAE-based approach such as [1].\n\n[1] Robust Imitation of Diverse Behaviors. Wang et al., NeurIPS 2017.\n\n**Detailed Comments:**\n\nJust to name a few grammar errors and typos:\n\n- \"it may not always generate\" → \"they may not always generate\"\n- \"help the model predicting\" → \"help the model predict\"\n- \"We use the target policy here because we take the outputs of the shared layers of the target network as a high level representation of the state and copy it to the style policy can make the optimization process of the style policy faster and more stable.\" grammar error", "summary_of_the_paper": "The paper proposes a method to learn different behavior styles from expert demonstrations via imitation learning. To achieve this, the method learns a latent style embedding of behavior styles and condition the policy on it. A style policy that takes the style embedding as additional input is learned alongside a target policy without the style embedding. Both policies are trained with behavior cloning (BC). Experiments are performed on simple environments such as Grid-World and Cartpole.", "summary_of_the_review": "In summary, the proposed idea lacks novelty and is similar to prior work [1]. Experiments are only performed on simple environments without proper comparison with all the baselines. There are also no ablation studies to understand the design of the method better. Thus, I vote to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4gerYhXGmf", "reviewer": "Reviewer_eS3s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strengths ===\n\n+ The paper is well written and the proposed idea is conveyed very clearly.\n+ The proposed method, StyleBC, is simple yet effective as it outperformed the selected strong baselines such as Info-GAIL in the CartPole environment. This is a plus. I am also very surprised that simply training the embedding with the highest prediction score works so well.\n\n=== Weaknesses ===\n\nMethod\n- I am not in favor of using a fixed sized (N) set embedding vectors, as this requires domain knowledge of the task.\n- My main concern with the proposed method is its core idea of using the style embeddings. Since it already uses a discrete set (N) of embedding vectors, why not simply use a conditional architecture, similar to Codevilla [1]? I would love to see experimental evidence if authors believe using discrete embedding is better than conditional branching.\n\nExperiments\n- There is no ablation study at all. I would like to see some ablation experiments that justify the design choice of StyleBC. For example, how would directly training the style policy compare to using a separate target policy for the state representation?\n- I wonder why there is no entry for Burn-InfoGAIL in table 1, as it is discussed in the related work that Burn-InfoGAIL should be preferred over InfoGAIL.\nThere is no comparison of StyleBC with baselines for the highway environment. There is no comparison with strong baselines in the gridworld environment.\n\n[1] End-to-end driving via conditional imitation learning, Codevilla et al., 2018\n", "summary_of_the_paper": "This paper presents Style BC, an extension to standard behavior cloning that uses an additional style embedding to capture different modes in expert demonstrations. The style embeddings are discrete, fixed-size vectors which the policy uses to predict actions from states. Style BC trains two imitative policies: the target policy and the style policy. The target policy directly predicts actions from states; the style policy additionally takes as input the selected style embedding. The style embeddings are trained by back-propagating action prediction gradients. Style BC outperforms the selected baselines (BC, GAIL, InfoGAIL) on both diversity and final performance in the CartPole environment. It is shown to be able to capture different modes from expert demonstrations in the Highway environment.\n\n", "summary_of_the_review": "The paper is well written and the proposed idea is well conveyed. However, there is no ablation study of the design choices, and overall experiments are not thorough. I do not recommend acceptance at this moment.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "21IFER5ZiE", "reviewer": "Reviewer_9MaF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Extending imitation learning frameworks taking into account the behavioural style is a very interesting topic with a potential of application in a variety of domain, from autonomous driving to HRI\n\n- From the reading, several aspects are not completely clear to me. I am not an expert on the problem, and I might have missed or misunderstood something, but I will try to summarise the concerts below:\n\n   * Not completely clear what do the authors mean saying that, with respect to behavioural cloning, their method tries to solve the problem in    \n     a more direct way (introduction)\n   * Not clear if style MDP is introduced by the authors or already existing in literature. Also I do not understand how the assumption of having \n     a behaviour policy as mixture of behaviour styles is reflected in the formulation (2.2)\n   * Not clear how the authors select the style vector c for each episode (3.1)\n   * What is in essence the role of the shared layers and the policy layers? (3.1). Overall, I find Fig. 2 a bit complicated, and not very well \n     contextualised with respect to the text (it does not always clarify as it should do)\n   * As the task of imitation is contextualised to an action classification task, is there a test set to be used? As the plots in Fig. 4 are on a \n     training set, they only provide a partial view of the picture and does speak about other important properties of the model, as the stability \n     and generalisation for instance (4.1.2)\n   * Although the experimental analysis includes different settings, I find it a bit limited\n\nAs a general comment, in some parts the language should be revised as it is a bit difficult to follow.\n", "summary_of_the_paper": "The paper is about imitation learning, and it particular it considers the task of replicating the style of behaviours. The authors propose a novel framework, called Style Behavioural Cloning that can learn the representation of different behaviour styles from demonstrations by experts. The approach is an extension of behaviour cloning, that tries to derive the behaviour style embedding by optimising the model to predict the action accurately.\nThe paper presents the mathematical formulation of the framework and an experimental analysis of different settings.", "summary_of_the_review": "Although I find the task very interesting and with a wide potential of application, I have a few concerns about the clarity of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers suggested rejection of the paper. This is based on concerns regarding novelty of results, clarity of presentation, simplicity of conducted experiments and missing ablation studies (and several other points raised in the reviews). The authors also did not submit a rebuttal. Hence I am recommending rejection of the paper.
Decision: Reject