Deep Imitative Models for Flexible Inference, Planning, and Control

Imitation Learning ( IL ) is an appealing approach to learn desirable autonomous behavior . However , directing IL to achieve arbitrary goals is difficult . In contrast , planning-based algorithms use dynamics models and reward functions to achieve goals . Yet , reward functions that evoke desirable behavior are often difficult to specify . In this paper , we propose “ Imitative Models ” to combine the benefits of IL and goal-directed planning . Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals . We derive families of flexible goal objectives , including constrained goal regions , unconstrained goal sets , and energy-based goals . We show that our method can use these objectives to successfully direct behavior . Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task , and is efficiently learned from expert demonstrations without online data collection . We also show our approach is robust to poorly specified goals , such as goals on the wrong side of the road . 1 INTRODUCTION . Imitation learning ( IL ) is a framework for learning a model to mimic behavior . At test-time , the model pursues its best-guess of desirable behavior . By letting the model choose its own behavior , we can not direct it to achieve different goals . While work has augmented IL with goal conditioning ( Dosovitskiy & Koltun , 2016 ; Codevilla et al. , 2018 ) , it requires goals to be specified during training , explicit goal labels , and are simple ( e.g. , turning ) . In contrast , we seek flexibility to achieve general goals for which we have no demonstrations . In contrast to IL , planning-based algorithms like model-based reinforcement learning ( MBRL ) methods do not require expert demonstrations . MBRL can adapt to new tasks specified through reward functions ( Kuvayev & Sutton , 1996 ; Deisenroth & Rasmussen , 2011 ) . The “ model ” is a dynamics model , used to plan under the user-supplied reward function . Planning enables these approaches to perform new tasks at test-time . The key drawback is that these models learn dynamics of possible behavior rather than dynamics of desirable behavior . This means that the responsibility of evoking desirable behavior is entirely deferred to engineering the input reward function . Designing reward functions that cause MBRL to evoke complex , desirable behavior is difficult when the space of possible undesirable behaviors is large . In order to succeed , the rewards can not lead the model astray towards observations significantly different than those with which the model was trained . Our goal is to devise an algorithm that combines the advantages of MBRL and IL by offering MBRL ’ s flexibility to achieve new tasks at test-time and IL ’ s potential to learn desirable behavior entirely from offline data . To accomplish this , we first train a model to forecast expert trajectories with a density function , which can score trajectories and plans by how likely they are to come from the expert . A probabilistic model is necessary because expert behavior is stochastic : e.g . at an intersection , the expert could choose to turn left or right . Next , we derive a principled probabilistic inference objective to create plans that incorporate both ( 1 ) the model and ( 2 ) arbitrary new tasks . Finally , we derive families of tasks that we can provide to the inference framework . Our method can accomplish new tasks specified as complex goals without having seen an expert complete these tasks before . We investigate properties of our method on a dynamic simulated autonomous driving task ( see Fig . 1 ) . Videos are available at https : //sites.google.com/view/imitative-models . Our contributions are as follows : 1 . Interpretable expert-like plans with minimal reward engineering . Our method outputs multistep expert-like plans , offering superior interpretability to one-step imitation learning models . In contrast to MBRL , our method generates expert-like behaviors with minimal reward engineering . 2 . Flexibility to new tasks : In contrast to IL , our method flexibly incorporates and achieves goals not seen during training , and performs complex tasks that were never demonstrated , such as navigating to goal regions and avoiding test-time only potholes , as depicted in Fig . 1 . 3 . Robustness to goal specification noise : We show that our method is robust to noise in the goal specification . In our application , we show that our agent can receive goals on the wrong side of the road , yet still navigate towards them while staying on the correct side of the road . 4 . State-of-the-art CARLA performance : Our method substantially outperforms MBRL , a custom IL method , and all five prior CARLA IL methods known to us . It learned near-perfect driving through dynamic and static CARLA environments from expert observations alone . 2 DEEP IMITATIVE MODELS . We begin by formalizing assumptions and notation . We model continuous-state , discrete-time , Partially-Observed Markov Decision Processes ( POMDPs ) . For brevity , we call the components of state of which we have direct observations the agent ’ s “ state ” , although we explicitly assume these states do not represent the full Markovian world state . Our agent ’ s state at time t is st ∈ RD ; t = 0 refers to the current time step , and φ is all of the agent ’ s observations . Variables are bolded . Random variables are capitalized . Absent subscripts denote all future time steps , e.g . S .= S1 : T ∈ RT×D . We denote a probability density function of a random variable S as p ( S ) , and its value as p ( s ) .= p ( S=s ) . To learn agent dynamics that are possible and preferred , we construct a model of expert behavior . We fit an “ Imitative Model ” q ( S1 : T |φ ) = ∏T t=1 q ( St|S1 : t−1 , φ ) to a dataset of expert trajectories D = { ( si , φi ) } Ni=1 drawn from a ( unknown ) distribution of expert behavior si ∼ p ( S|φi ) . By training q ( S|φ ) to forecast expert trajectories with high likelihood , we model the scene-conditioned expert dynamics , which can score trajectories by how likely they are to come from the expert . 2.1 IMITATIVE PLANNING TO GOALS . After training , q ( S|φ ) can generate trajectories that resemble those that the expert might generate – e.g . trajectories that navigate roads with expert-like maneuvers . However , these maneuvers will not have a specific goal . Beyond generating human-like behaviors , we wish to direct our agent to goals and have the agent automatically reason about the necessary mid-level details . We define general tasks by a set of goal variables G. The probability of a plan s conditioned on the goal G is modelled by a posterior p ( s|G , φ ) . This posterior is implemented with q ( s|φ ) as a learned imitation prior and p ( G|s , φ ) as a test-time goal likelihood . We give examples of p ( G|s , φ ) after deriving a maximum a posteriori inference procedure to generate expert-like plans that achieve abstract goals : s∗ . = arg max s log p ( s|G , φ ) = arg max s log q ( s|φ ) + log p ( G|s , φ ) − log p ( G|φ ) = arg max s log q ( s|φ ) ︸ ︷︷ ︸ imitation prior + log p ( G|s , φ ) ︸ ︷︷ ︸ goal likelihood . ( 1 ) We perform gradient-based optimization of Eq . 1 , and defer this discussion to Appendix A . Next , we discuss several goal likelihoods , which direct the planning in different ways . They communicate goals they desire the agent to achieve , but not how to achieve them . The planning procedure determines how to achieve them by producing paths similar to those an expert would have taken to reach the given goal . In contrast to black-box one-step IL that predicts controls , our method produces interpretable multi-step plans accompanied by two scores . One estimates the plan ’ s “ expertness ” , the second estimates its probability to achieve the goal . Their sum communicates the plan ’ s overall quality . Our approach can also be viewed as a learning-based method to integrate mid-level and high-level controllers together , where demonstrations from both are available at train-time , only the highlevel controller is available at test-time , and the high-level controller can vary . The high-level controller ’ s action specifies a subgoal for the mid-level controller . A density model of future trajectories of an expert mid-level controller is learned at train-time , and is amenable to different types of direction as specified by the high-level controller . In this sense , the model is an “ apprentice ” , having learned to imitate mid-level behaviors . In our application , the high-level controller is composed of an A∗ path-planning algorithm and one of a library of components that forms goal likelihoods from the waypoints produced by A∗ . Connecting this to related approaches , learning the midlevel controller ( Imitative Model ) resembles offline IL , whereas inference with an Imitative Model resembles trajectory optimization in MBRL , given goals provided by the high-level controller . 2.2 CONSTRUCTING GOAL LIKELIHOODS . Constraint-based planning to goal sets ( hyperparameter-free ) : Consider the setting where we have access to a set of desired final states , one of which the agent should achieve . We can model this by applying a Dirac-delta distribution on the final state , to ensure it lands in a goal set G ⊂RD : p ( G|s , φ ) ← δsT ( G ) , δsT ( G ) = 1 if sT ∈ G , δsT ( G ) = 0 if sT 6∈ G. ( 2 ) δsT ( G ) ’ s partial support of sT ∈ G ⊂RD constrains sT and introduces no hyperparameters into p ( G|s , φ ) . For each choice of G , we have a different way to provide high-level task information to the agent . The simplest choice for G is a finite set of points : a ( A ) Final-State Indicator likelihood . We applied ( A ) to a sequence of waypoints received from a standard A∗ planner ( provided by the CARLA simulator ) , and outperformed all prior dynamic-world CARLA methods known to us . We can also consider providing an infinite number of points . Providing a set of line-segments as G yields a ( B ) Line-Segment Final-State Indicator likelihood , which encourages the final state to land along one of the segments . Finally , consider a ( C ) Region Final-State Indicator likelihood in which G is a polygon ( see Figs . 1 and 4 ) . Solving Eq . 1 with ( C ) amounts to planning the most expert-like trajectory that ends inside a goal region . Appendix B provides derivations , implementation details , and additional visualizations . We found these methods to work well when G contains “ expert-like ” final position ( s ) , as the prior strongly penalizes plans ending in non-expert-like positions . Unconstrained planning to goal sets ( hyperparameter-based ) : Instead of constraining that the final state of the trajectory reach a goal , we can use a goal likelihood with full support ( sT ∈RD ) , centered at a desired final state . This lets the goal likelihood encourage goals , rather than dictate them . If there is a single desired goal ( G= { gT } ) , the ( D ) Gaussian Final-State likelihood p ( G|s , φ ) ← N ( gT ; sT , I ) treats gT as a noisy observation of a final future state , and encourages the plan to arrive at a final state . We can also plan to K successive states G = ( gT−K+1 , . . . , gT ) with a ( E ) Gaussian State Sequence : p ( G|s , φ ) ← ∏T k=T−K+1N ( gk ; sk , I ) if a program wishes to specify a desired end velocity or acceleration when reaching the final state gT ( Fig . 2 ) . Alternatively , a planner may propose a set of states with the intention that the agent should reach any one of them . This is possible by using a ( F ) Gaussian Final-State Mixture : p ( G|s , φ ) ← 1K ∑K k=1N ( gkT ; sT , I ) and is useful if some of those final states are not reachable with an expert-like plan . Unlike A–C , D–F introduce a hyperparameter “ ” . However , they are useful when no states in G correspond to observed expert behavior , as they allow the imitation prior to be robust to poorly specified goals . Costed planning : Our model has the additional flexibility to accept arbitrary user-specified costs c at test-time . For example , we may have updated knowledge of new hazards at test-time , such as a given map of potholes or a predicted cost map . Cost-based knowledge c ( si|φ ) can be incorporated as an ( G ) Energy-based likelihood : p ( G|s , φ ) ∝ ∏T t=1 e −c ( st|φ ) ( Todorov , 2007 ; Levine , 2018 ) . This can be combined with other goal-seeking objectives by simply multiplying the likelihoods together . Examples of combining G ( energy-based ) with F ( Gaussian mixture ) were shown in Fig . 1 and are shown in Fig . 3 . Next , we describe instantiating q ( S|φ ) in CARLA ( Dosovitskiy et al. , 2017 ) . Designing general goal likelihoods can be considered a form of reward engineering if there are no restrictions on the goal likelihoods . This connection is best seen in ( G ) , which has an explicit cost term . One reason why it is easier to design goal likelihoods than to design reward functions is that the task of evoking most aspects of goal-driven behavior is already learned by the prior q ( s|φ ) , which models desirable behavior . This is in contrast to model-free RL , which entirely relies on the reward design to evoke goal-driven behavior , and in contrast to model-based RL , which heavily relies on the reward design to evoke goal-driven behavior , as its dynamics model learns what is possible , rather than what is desirable . Additionally , it is easy to design goal likelihoods when goals provide a significant amount of information that obviates the need to do any manual tuning . The main assumption is that one of the goals in the goal set is reachable within the model ’ s time-horizon .

Summary: The paper takes a model-based approach to "imitation learning". It first learns a (flow) "model" to assign likelihoods to trajectories being expert-like as well as sample expert-like trajectories. This is then combined with an estimate for trajectories being certain goal conditioned, where the goals come from a route planner. Results are shown in the context of autonomous driving in the CARLA simulator with a PID controller tracking the open-loop plan from the planner.