Deep Imitative Models for Flexible Inference, Planning, and Control

Imitation Learning ( IL ) is an appealing approach to learn desirable autonomous behavior . However , directing IL to achieve arbitrary goals is difficult . In contrast , planning-based algorithms use dynamics models and reward functions to achieve goals . Yet , reward functions that evoke desirable behavior are often difficult to specify . In this paper , we propose “ Imitative Models ” to combine the benefits of IL and goal-directed planning . Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals . We derive families of flexible goal objectives , including constrained goal regions , unconstrained goal sets , and energy-based goals . We show that our method can use these objectives to successfully direct behavior . Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task , and is efficiently learned from expert demonstrations without online data collection . We also show our approach is robust to poorly specified goals , such as goals on the wrong side of the road . 1 INTRODUCTION . Imitation learning ( IL ) is a framework for learning a model to mimic behavior . At test-time , the model pursues its best-guess of desirable behavior . By letting the model choose its own behavior , we can not direct it to achieve different goals . While work has augmented IL with goal conditioning ( Dosovitskiy & Koltun , 2016 ; Codevilla et al. , 2018 ) , it requires goals to be specified during training , explicit goal labels , and are simple ( e.g. , turning ) . In contrast , we seek flexibility to achieve general goals for which we have no demonstrations . In contrast to IL , planning-based algorithms like model-based reinforcement learning ( MBRL ) methods do not require expert demonstrations . MBRL can adapt to new tasks specified through reward functions ( Kuvayev & Sutton , 1996 ; Deisenroth & Rasmussen , 2011 ) . The “ model ” is a dynamics model , used to plan under the user-supplied reward function . Planning enables these approaches to perform new tasks at test-time . The key drawback is that these models learn dynamics of possible behavior rather than dynamics of desirable behavior . This means that the responsibility of evoking desirable behavior is entirely deferred to engineering the input reward function . Designing reward functions that cause MBRL to evoke complex , desirable behavior is difficult when the space of possible undesirable behaviors is large . In order to succeed , the rewards can not lead the model astray towards observations significantly different than those with which the model was trained . Our goal is to devise an algorithm that combines the advantages of MBRL and IL by offering MBRL ’ s flexibility to achieve new tasks at test-time and IL ’ s potential to learn desirable behavior entirely from offline data . To accomplish this , we first train a model to forecast expert trajectories with a density function , which can score trajectories and plans by how likely they are to come from the expert . A probabilistic model is necessary because expert behavior is stochastic : e.g . at an intersection , the expert could choose to turn left or right . Next , we derive a principled probabilistic inference objective to create plans that incorporate both ( 1 ) the model and ( 2 ) arbitrary new tasks . Finally , we derive families of tasks that we can provide to the inference framework . Our method can accomplish new tasks specified as complex goals without having seen an expert complete these tasks before . We investigate properties of our method on a dynamic simulated autonomous driving task ( see Fig . 1 ) . Videos are available at https : //sites.google.com/view/imitative-models . Our contributions are as follows : 1 . Interpretable expert-like plans with minimal reward engineering . Our method outputs multistep expert-like plans , offering superior interpretability to one-step imitation learning models . In contrast to MBRL , our method generates expert-like behaviors with minimal reward engineering . 2 . Flexibility to new tasks : In contrast to IL , our method flexibly incorporates and achieves goals not seen during training , and performs complex tasks that were never demonstrated , such as navigating to goal regions and avoiding test-time only potholes , as depicted in Fig . 1 . 3 . Robustness to goal specification noise : We show that our method is robust to noise in the goal specification . In our application , we show that our agent can receive goals on the wrong side of the road , yet still navigate towards them while staying on the correct side of the road . 4 . State-of-the-art CARLA performance : Our method substantially outperforms MBRL , a custom IL method , and all five prior CARLA IL methods known to us . It learned near-perfect driving through dynamic and static CARLA environments from expert observations alone . 2 DEEP IMITATIVE MODELS . We begin by formalizing assumptions and notation . We model continuous-state , discrete-time , Partially-Observed Markov Decision Processes ( POMDPs ) . For brevity , we call the components of state of which we have direct observations the agent ’ s “ state ” , although we explicitly assume these states do not represent the full Markovian world state . Our agent ’ s state at time t is st ∈ RD ; t = 0 refers to the current time step , and φ is all of the agent ’ s observations . Variables are bolded . Random variables are capitalized . Absent subscripts denote all future time steps , e.g . S .= S1 : T ∈ RT×D . We denote a probability density function of a random variable S as p ( S ) , and its value as p ( s ) .= p ( S=s ) . To learn agent dynamics that are possible and preferred , we construct a model of expert behavior . We fit an “ Imitative Model ” q ( S1 : T |φ ) = ∏T t=1 q ( St|S1 : t−1 , φ ) to a dataset of expert trajectories D = { ( si , φi ) } Ni=1 drawn from a ( unknown ) distribution of expert behavior si ∼ p ( S|φi ) . By training q ( S|φ ) to forecast expert trajectories with high likelihood , we model the scene-conditioned expert dynamics , which can score trajectories by how likely they are to come from the expert . 2.1 IMITATIVE PLANNING TO GOALS . After training , q ( S|φ ) can generate trajectories that resemble those that the expert might generate – e.g . trajectories that navigate roads with expert-like maneuvers . However , these maneuvers will not have a specific goal . Beyond generating human-like behaviors , we wish to direct our agent to goals and have the agent automatically reason about the necessary mid-level details . We define general tasks by a set of goal variables G. The probability of a plan s conditioned on the goal G is modelled by a posterior p ( s|G , φ ) . This posterior is implemented with q ( s|φ ) as a learned imitation prior and p ( G|s , φ ) as a test-time goal likelihood . We give examples of p ( G|s , φ ) after deriving a maximum a posteriori inference procedure to generate expert-like plans that achieve abstract goals : s∗ . = arg max s log p ( s|G , φ ) = arg max s log q ( s|φ ) + log p ( G|s , φ ) − log p ( G|φ ) = arg max s log q ( s|φ ) ︸ ︷︷ ︸ imitation prior + log p ( G|s , φ ) ︸ ︷︷ ︸ goal likelihood . ( 1 ) We perform gradient-based optimization of Eq . 1 , and defer this discussion to Appendix A . Next , we discuss several goal likelihoods , which direct the planning in different ways . They communicate goals they desire the agent to achieve , but not how to achieve them . The planning procedure determines how to achieve them by producing paths similar to those an expert would have taken to reach the given goal . In contrast to black-box one-step IL that predicts controls , our method produces interpretable multi-step plans accompanied by two scores . One estimates the plan ’ s “ expertness ” , the second estimates its probability to achieve the goal . Their sum communicates the plan ’ s overall quality . Our approach can also be viewed as a learning-based method to integrate mid-level and high-level controllers together , where demonstrations from both are available at train-time , only the highlevel controller is available at test-time , and the high-level controller can vary . The high-level controller ’ s action specifies a subgoal for the mid-level controller . A density model of future trajectories of an expert mid-level controller is learned at train-time , and is amenable to different types of direction as specified by the high-level controller . In this sense , the model is an “ apprentice ” , having learned to imitate mid-level behaviors . In our application , the high-level controller is composed of an A∗ path-planning algorithm and one of a library of components that forms goal likelihoods from the waypoints produced by A∗ . Connecting this to related approaches , learning the midlevel controller ( Imitative Model ) resembles offline IL , whereas inference with an Imitative Model resembles trajectory optimization in MBRL , given goals provided by the high-level controller . 2.2 CONSTRUCTING GOAL LIKELIHOODS . Constraint-based planning to goal sets ( hyperparameter-free ) : Consider the setting where we have access to a set of desired final states , one of which the agent should achieve . We can model this by applying a Dirac-delta distribution on the final state , to ensure it lands in a goal set G ⊂RD : p ( G|s , φ ) ← δsT ( G ) , δsT ( G ) = 1 if sT ∈ G , δsT ( G ) = 0 if sT 6∈ G. ( 2 ) δsT ( G ) ’ s partial support of sT ∈ G ⊂RD constrains sT and introduces no hyperparameters into p ( G|s , φ ) . For each choice of G , we have a different way to provide high-level task information to the agent . The simplest choice for G is a finite set of points : a ( A ) Final-State Indicator likelihood . We applied ( A ) to a sequence of waypoints received from a standard A∗ planner ( provided by the CARLA simulator ) , and outperformed all prior dynamic-world CARLA methods known to us . We can also consider providing an infinite number of points . Providing a set of line-segments as G yields a ( B ) Line-Segment Final-State Indicator likelihood , which encourages the final state to land along one of the segments . Finally , consider a ( C ) Region Final-State Indicator likelihood in which G is a polygon ( see Figs . 1 and 4 ) . Solving Eq . 1 with ( C ) amounts to planning the most expert-like trajectory that ends inside a goal region . Appendix B provides derivations , implementation details , and additional visualizations . We found these methods to work well when G contains “ expert-like ” final position ( s ) , as the prior strongly penalizes plans ending in non-expert-like positions . Unconstrained planning to goal sets ( hyperparameter-based ) : Instead of constraining that the final state of the trajectory reach a goal , we can use a goal likelihood with full support ( sT ∈RD ) , centered at a desired final state . This lets the goal likelihood encourage goals , rather than dictate them . If there is a single desired goal ( G= { gT } ) , the ( D ) Gaussian Final-State likelihood p ( G|s , φ ) ← N ( gT ; sT , I ) treats gT as a noisy observation of a final future state , and encourages the plan to arrive at a final state . We can also plan to K successive states G = ( gT−K+1 , . . . , gT ) with a ( E ) Gaussian State Sequence : p ( G|s , φ ) ← ∏T k=T−K+1N ( gk ; sk , I ) if a program wishes to specify a desired end velocity or acceleration when reaching the final state gT ( Fig . 2 ) . Alternatively , a planner may propose a set of states with the intention that the agent should reach any one of them . This is possible by using a ( F ) Gaussian Final-State Mixture : p ( G|s , φ ) ← 1K ∑K k=1N ( gkT ; sT , I ) and is useful if some of those final states are not reachable with an expert-like plan . Unlike A–C , D–F introduce a hyperparameter “ ” . However , they are useful when no states in G correspond to observed expert behavior , as they allow the imitation prior to be robust to poorly specified goals . Costed planning : Our model has the additional flexibility to accept arbitrary user-specified costs c at test-time . For example , we may have updated knowledge of new hazards at test-time , such as a given map of potholes or a predicted cost map . Cost-based knowledge c ( si|φ ) can be incorporated as an ( G ) Energy-based likelihood : p ( G|s , φ ) ∝ ∏T t=1 e −c ( st|φ ) ( Todorov , 2007 ; Levine , 2018 ) . This can be combined with other goal-seeking objectives by simply multiplying the likelihoods together . Examples of combining G ( energy-based ) with F ( Gaussian mixture ) were shown in Fig . 1 and are shown in Fig . 3 . Next , we describe instantiating q ( S|φ ) in CARLA ( Dosovitskiy et al. , 2017 ) . Designing general goal likelihoods can be considered a form of reward engineering if there are no restrictions on the goal likelihoods . This connection is best seen in ( G ) , which has an explicit cost term . One reason why it is easier to design goal likelihoods than to design reward functions is that the task of evoking most aspects of goal-driven behavior is already learned by the prior q ( s|φ ) , which models desirable behavior . This is in contrast to model-free RL , which entirely relies on the reward design to evoke goal-driven behavior , and in contrast to model-based RL , which heavily relies on the reward design to evoke goal-driven behavior , as its dynamics model learns what is possible , rather than what is desirable . Additionally , it is easy to design goal likelihoods when goals provide a significant amount of information that obviates the need to do any manual tuning . The main assumption is that one of the goals in the goal set is reachable within the model ’ s time-horizon .

Summary: This submission proposes a method to combine the benefits of model-based RL and Imitation Learning (IL) for navigation tasks. The key idea is to i) learn a prior over trajectory distributions from a fixed dataset of demonstrations, and ii) use this learned dynamical model for path planning via probabilistic inference. Reaching target waypoints is done by maximizing the trajectory likelihood conditioned on the planning goal. The prior is learned using R2P2 on LIDAR features and past positions. Experiments using the CARLA driving simulator show that this method can outperform standard control, IL, and model-based RL baselines, while flexibly incorporating test-time goals and costs thanks to its probabilistic formulation.