Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

1 INTRODUCTION . Complex human activities can often be broken down into various simpler sub-activities or sub-tasks that can serve as the basic building blocks for completing a variety of complicated tasks . For instance , when driving a car , a driver may perform several simpler sub-tasks such as driving straight in a lane , changing lanes , executing a turn and braking , in different orders and for varying times depending on the source , destination , traffic conditions etc . Using imitation learning to learn a single monolithic policy to represent a structured activity can be challenging as it does not make explicit the sub-structure between the parts within the activity . In this work , we develop an imitation learning framework that can learn a policy for each of these sub-tasks given unsegmented activity demonstrations and also learn a macro-policy which dictates switching from one sub-task policy to another . Learning sub-task specific policies has the benefit of shared learning . Each such sub-task policy also needs to specialize over a restricted state space , thus making the learning problem easier . Previous works in imitation learning ( Li et al. , 2017 ; Hausman et al. , 2017 ) focus on learning each sub-task specific policy using segmented expert demonstrations by modeling the variability in each sub-task policy using a latent variable . This latent variable is inferred by enforcing high mutual information between the latent variable and expert demonstrations . This information theoretic perspective is equivalent to the graphical model shown in Figure 1 ( Left ) , where the node c represents the latent variable . However , since learning sub-task policies requires isolated demonstrations for each sub-task , this setup is difficult to scale to many real world scenarios where providing such segmented trajectories is cumbersome . Further , this setup does not learn a macro-policy to combine the learned sub-task policies in meaningful ways to achieve different tasks . In our work , we aim to learn each sub-task policy directly from unsegmented activity demonstrations . For example , given a task consisting of three sub-tasks — A , B and C , we wish to learn a policy to complete sub-task A , learn when to transition from A to B , finish sub-task B and so on . To achieve this we use a causal graphical model , which can be represented as a Dynamic Bayesian Network as ∗Denotes equal contribution shown in Figure 1 ( Right ) . The nodes ct denote latent variables which indicate the currently active sub-task and the nodes τt denote the state-action pair at time t. We consider as given , a set of expert demonstrations , each of which is represented by τ = { τ1 , · · · , τT } and has a corresponding sequence of latent factors c = { c1 , · · · , cT−1 } . The sub-activity at time t dictates what state-action pair was generated at time t. The previous sub-task and the current state together cause the selection of the next sub-task . As we will discuss in Section 3 , extending the use of mutual information to learn sub-task policies from unsegmented demonstrations is problematic , as it requires learning the macro-policy as a conditional probability distribution which depends on the unobserved future . This unobserved future is unknown during earlier points of interaction ( Figure 1 ) . To alleviate this , in our work we aim to force the policy to generate trajectories that maximize the directed information or causal information ( Massey , 1990 ) flow from trajectories to latent factors of variation within the trajectories instead of mutual information . Using directed information requires us to learn a causally conditioned probability distribution ( Kramer , 1998 ) which depends only on the observed past while allowing the unobserved future to be sequentially revealed . Further , since there exists feedback in our causal graphical model i.e. , information flows from the latent variables to trajectories and vice versa , directed information also provides a better upper bound on this information flow between the latent variables and expert trajectories than does the conventional mutual information ( Massey , 1990 ; Kramer , 1998 ) . We also draw connections with existing work on learning sub-task policies using imitation learning with the options framework ( Sutton et al. , 1998 ; Daniel et al. , 2016 ) . We show that our work , while derived using the information theoretic perspective of maximizing directed information , bears a close resemblance to applying the options framework in a generative adversarial imitation setting . Thus , our approach combines the benefits of learning hierarchical policies using the options framework with the robustness of generative adversarial imitation learning , helping overcome problems such as compounding errors that plague behaviour cloning . In summary , the main contributions of our work include : • We extend existing generative adversarial imitation learning frameworks to allow for learning of sub-task specific policies by maximizing directed information in a causal graph of subactivity latent variables and observed trajectory variables . • We draw connections between previous works on imitation learning with sub-task policies using options and show that our proposed approach can also be seen as option learning in a generative adversarial setting . • We show through experiments on both discrete and continuous state-action spaces , the ability of our approach to segment expert demonstrations into meaningful sub-tasks and combine sub-task specific policies to perform the desired task . 2 RELATED WORK . 2.1 IMITATION LEARNING . Imitation Learning ( Pomerleau , 1989 ) aims at learning policies that can mimic expert behaviours from demonstrations . Modeling the problem as a Markov Decision Process ( MDP ) , the goal in imitation learning is to learn a policy π ( a|s ) , which defines the conditional distribution over actions a ∈ A given the state s ∈ S , from state-action trajectories τ = ( s0 , a0 , · · · , sT ) of expert behaviour . Recently , Ho & Ermon ( 2016 ) introduced an imitation learning framework called Generative Adversarial Imitation Learning ( GAIL ) that is able to learn policies for complex high-dimensional physics-based control tasks . They reduce the imitation learning problem into an adversarial learning framework , for which they utilize Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) . The generator network of the GAN represents the agent ’ s policy π while the discriminator network serves as a local reward function and learns to differentiate between state-action pairs from the expert policy πE and from the agent ’ s policy π . Mathematically , it is equivalent to optimizing the following , min π max D Eπ [ logD ( s , a ) ] + EπE [ 1− logD ( s , a ) ] − λH ( π ) InfoGAIL ( Li et al. , 2017 ) and Hausman et al . ( 2017 ) solve the problem of learning from policies generated by a mixture of experts . They introduce a latent variable c into the policy function π ( a|s , c ) to separate different type of behaviours present in the demonstration . To incentivize the network to use the latent variable , they utilize an information-theoretic regularization enforcing that there should be high mutual information between c and the state-action pairs in the generated trajectory , a concept that was first introduced in InfoGAN ( Chen et al. , 2016 ) . They introduce a variational lower bound L1 ( π , Q ) of the mutual information I ( c ; τ ) to the loss function in GAIL . L1 ( π , Q ) = Ec∼p ( c ) , a∼π ( ·|s , c ) logQ ( c|τ ) +H ( c ) ≤ I ( c ; τ ) The modified objective can then be given as , min π , q max D Eπ [ logD ( s , a ) ] + EπE [ 1− logD ( s , a ) ] − λ1L1 ( π , q ) − λ2H ( π ) InfoGAIL models variations between different trajectories as the latent codes correspond to trajectories coming from different demonstrators . In contrast , we aim to model intra-trajectory variations and latent codes in our work correspond to sub-tasks ( variations ) within a demonstration . In Section 3 , we discuss why using a mutual information based loss is infeasible in our problem setting and describe our proposed approach . 2.2 OPTIONS . Consider an MDP with states s ∈ S and actions a ∈ A . Under the options framework ( Sutton et al. , 1998 ) , an option , indexed by o ∈ O consists of a sub-policy π ( a|s , o ) , a termination policy π ( b|s , ō ) and an option activation policy π ( o|s ) . After an option is initiated , actions are generated by the sub-policy until the option is terminated and a new option is selected . Options framework has been studied widely in RL literature . A challenging problem related to the options framework is to automatically infer options without supervision . Option discovery approaches often aim to find bottleneck states , i.e. , states that the agent has to pass through to reach the goal . Many different approaches such as multiple-instance learning ( McGovern & Barto , 2001 ) , graph based algorithms ( Menache et al. , 2002 ; Şimşek et al. , 2005 ) have been used to find such bottleneck states . Once the bottleneck states are discovered , the above approaches find options policies to reach each such state . In contrast , we propose a unified framework using a information-theoretic approach to automatically discover relevant option policies without the need to discover bottleneck states . Daniel et al . ( 2016 ) formulate the options framework as a probabilistic graphical model where options are treated as latent variables which are then learned from expert data . The option policies ( π ( a|s , o ) ) are analogous to sub-task policies in our work . These option policies are then learned by maximizing a lower bound using the Expectation-Maximization algorithm ( Moon , 1996 ) . We show how this lower bound is closely related to the objective derived in our work . We further show how this connection allows our method to be seen as a generative adversarial variant of their approach . Fox et al . ( 2017 ) propose to extend the EM based approach to multiple levels of option hierarchies . Further work on discovery of deep continuous options ( Krishnan et al. , 2017 ) allows the option policy to also select a continuous action in states where none of the options are applicable . Our proposed approach can also be extended to multi-level hierarchies ( e.g . by learning VAEs introduced in section 3 with multiple sampling layers ) or hybrid categorical-continuous macro-policies ( e.g . using both categorical and continuous hidden units in the sampling layer in VAE ) . Shiarlis et al . ( 2018 ) learn options by assuming knowledge of task sketches ( Andreas et al. , 2017 ) along with the demonstrations . The work proposes a behavior cloning based approach using connectionist temporal classification ( Graves et al. , 2006 ) to simultaneously maximize the joint likelihood of the sketch sequences and the sub-policies . Our proposed approach does not expect task sketches as input , making it more amenable to problems where labeling demonstrations with sketch labels is difficult . Prior work in robot learning has also looked at learning motion primitives from unsegmented demonstrations . These primitives usually correspond to a particular skill and are analogous to options . Niekum & Barto ( 2011 ) used the Beta-Process Autoregressive Hidden Markov Model ( BP-AR-HMM ) to segment expert demonstrations and post-process these segments to learn motion primitives which provide the ability to use reinforcement learning for policy improvement . Alternately , Krishnan et al . ( 2018 ) use Dirichlet Process Gaussian Mixture Model ( DP-GMM ) to segment the expert demonstrations by finding transition states between linear dynamical segments . Similarly , Ranchod et al . ( 2015 ) use the BP-AR-HMM framework to initially segment the expert demonstrations and then use an inverse reinforcement learning step to infer the reward function for each segment . The use of appropriate priors allows these methods to discover options without a priori knowledge of the total number of skills . Kroemer et al . ( 2014 ) model the task of manipulation as an autoregressive Hidden Markov Model where the hidden phases of manipulation are learned from data using EM . However , unlike the above methods , in our proposed approach we also learn an appropriate policy over the extracted options . We show how this allows us to compose the individual option policies to induce novel behaviours which were not present in the expert demonstrations .

Summary: This paper proposes an extension over the popular GAIL method for imitation learning  for the multi-modal data or tasks that have hierarchical structure in them. To achieve that the paper introduces an unsupervised variational objective by maximizing the directed mutual information between the latents c’s and the trajectories. The advantage of using directed information instead of regular MI based criterion is two-folds: 1) Being able to express the causal and temporal dependencies among the c’s changing across time. 2) Being able to learn a macro-policy without needing to condition on the future trajectories. Authors present results both on continuous and discrete environments.