Deep Imitative Models for Flexible Inference, Planning, and Control
Authors: Nicholas Rhinehart
Rowan McAllister
Sergey Levine
Venue:  (2020)
Keywords: imitation learning
planning
autonomous driving

Abstract: Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose "Imitative Models" to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection.  We also show our approach is robust to poorly-specified goals, such as goals on the wrong side of the road.

Published as a conference paper at ICLR 2020
DEEP IMITATIVE MODELS FOR FLEXIBLE INFERENCE, PLANNING, AND CONTROL
Nicholas Rhinchart UC Berkeley arbinchart@berkeley edu
Rowan McAllister UC Berkeley rmcallister@berkeley edu
Sergey Levine UC Berkeley svlevine swlevine@berkeley.
ABSTRACT
Imitation Learning (II.) is an appealing approach learn desirable autonomous behavior. However, directing IL achieve arbitrary goals is difficult. In contrast, planning-base algorithms use dynamics models and reward functions to achieve goals. Yet, reward funetions that evoke desirable behavior often difficult specify. In this paper, we propose "Imitative Models" to combine the benefits of IL and goal-directed planning. Imitative Models probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning- -based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.
INTRODUCTION
Imitation learning (IL) a framework for learning a model mimic behavior. At test-time, the model pursues its best-guess of desirable behavior. By letting the model choose its own behavior, we cannot direct to achieve different goals. While work has augmented II. with goal conditioning (Dosovitskiy & Koltun, 2016; Codevilla et al., 2018), it requires goals to be specified during training, explicit goal labels, and are simple (e.g., turning). In contrast, we seek flexibility to achieve general goals for which we have no demonstrations.
In contrast to II, planning- based algorithms like model-based reinforcement learning (MBRL) methods do not require expert demonstrations. MBRL adapt to new tasks specified through reward functions (Kuvayev & Sutton, 1996; Deisenroth Rasmussen, 2011). The "model" is a dynamics model, used to plan under the user-supplied reward function. Planning enables these approaches to perform new tasks at test-time. The key drawback is that these models learn dynamics of possible behavior rather than dynamics of desirable behavior. This means that the responsibility of evoking desirable behavior is entirely deferred to engineering the input reward function. Designing reward functions that cause MBRL evoke complex, desirable behavior is difficult when the space of possible undesirable behaviors is large. In order to succeed, the rewards cannot lead the model astray towards observations significantly different than those with which the model was trained.
Our goal is to devise an algorithm that combines the ad advantages of MBRL and IL by offering MBRL's flexibility to achieve new tasks at test-time and II's potential to learn desirable behavior entirely from offline data. To accomplish this, we first train a model to forecast expert trajectories with density function, which can score trajectories and plans by how likely they are to come from the expert. A probabilistic model is necessary because expert behavior is stochastic: e.g. at an intersection, the expert could choose to turn left or right. Next, we derive a principled probabilistic inference objective tasks. Finally, derive
to create plans that incorporate both (1) the model and (2) arbitrary families of tasks that we can provide to the inference framework. Our method can accomplish new tasks specified as complex goals without having seen expert complete these tasks before.

---

Published as a conference paper at ICLR 2020
Training Data: (Seone 9. Humnnn Fature *yy) pairs
Beginn Gaa Took Go sireerhee thk
Potet Goals and Pecboke Thci
Potholes
Goals
Imitative Model
toxiszaeed prodedillay donnity Gokar sxpert lelanin)
((817(0)
would Immaa these golde while aoviding poelicos?
region go?
Potholes
Hest plan BR
Goals
Plans
LIDAR Scon
Gool Plan grebevaee fhe Ligk
Figure Our method: deep imitative models. Top Center. We use demonstrations learn probability density function q of future behavior and deploy it to accomplish various tasks. Left: A region the ground plane is input achieve that task. It coarsely specifies a destination, and guides the vehicle to turn left. Right: Goal positions and potholes yield a plan that avoids potholes and achieves one of the goals on the right.
planning procedure that reasons about how the expert would
We investigate properties of our method on a dynamic simulated autonomous driving task (see Fig. 1). Vidcos are available at https://sites wesiieessgogllee.. com/cies////oeeeeccoo http//eewwiiiiaaaeeeeeeeeee iwwiittiiiiiieeiieeiiieeeiiiitie models... Our contributions are as follows:
1. Interpretable expert-like plans with minimal reward engineering, Our method outputs multi- step expert-like plans, offering superior interpretability to one-step imitation learning models. In contrast to MBRL, our method generates expert-like behaviors with minimal reward engineering,
Flexibility new tasks: contrast to IL, our method flexibly incorporates and achieves goals not seen during training, and performs complex tasks that were never demonstrated, such as navigating to goal regions and avoiding test-time only potholes, as depicted Fig. 1.
3. Robustness to goal specification noise: We show that our method is robust to noise in the goal specification. In our application, we show that our agent can receive goals on the wrong side of the road, yet still navigate towards them while staying the correct side of the road.
4. -tate-of-the-art CA CARLA performance: Our method substantially outperforms MBRL, a custom IL method, and all five prior CARLA IL methods known us. It learned near-perfect driving through dynamic and static CARLA environments from expert observations alone.
2 DEEP IMITATIVE MODELS
We begin by formalizing assumptions and notation. We model continuous-state, discrete-time, Partially-Observed Markov Decision Processes (POMDPs). For brevity, we call the components of state of which we have direct observations the agent's "state". although we explicitly assume these states do not represent the full Markovian world state. Our agent's 's state at time t is S; E IR t 0 refers the current time step, and @ all of the agent's observations. Variables are bolded. Random variables are capitalized. Absent subscripts denote all future time steps, e.g. S = S1.7 E R We denote a probability density function of random variable S as pS), and its value as p(s) = p( sSs
To learn agent dynamics that are possible and preferred, we construct model of expert behavior. We fit an "Imitative Model" q(Si.r (p) IT= / Φ) dataset of expert trajectories D o')) drawn from a (unknown) distribution of expert behavior s' p(S Ø²). By training g(S Φ) to forecast expert trajectories with high likelihood, we model the scene-conditiones expert dynamics, which can score trajectories by how likely they come from the expert.
2

---

Published as a conference paper at ICLR 2020
2.1 IMITATIVE PLANNING TO GOALS
After training, (((()) can generate trajectories that resemble those that the expert might generate- e.g. trajectories that navigate roads with expert-like maneuvers. However, these maneuvers will not have a specific goal. Beyond generating human-like behaviors, we wish to direct our agent to goals and have the agent automatically reason about the necessary mid-level details. We define general tasks by set of goal variables G. The probability of a plan s conditioned on the goal G is modelled by a posterior p( This posterior implemented with q(s) as a learned imitation prior and p(G|s, Φ) test-time goal likelihood. We give examples of p(9 (s, Φ) after deriving maximum posteriori inference procedure to generate expert-like plans that achieve abstract goals:
s' = arg max log Dogp(s(g.c) Φ) arg max log ((| + log p(g)s o) log (()
arg arg max log ((ا + log p(g Ø).
imitation prior
goal likelihood
We perform gradient- -ased optimization of Eq. 1, and defer this discussion to Appendix A. Next, we discuss several goal likelihoods, which direct the planning in different ways. They communicate goals they desire the agent to achieve, but not how to achieve them. The planning procedure determines how to achieve them by producing paths similar to those an expert would have taken to reach the given goal. In contrast to black-box one-step II. that predicts controls, our method produces interpretable multi-step plans accompanied by two scores. One estimates the plan's 's "expertness", the second estimates its probability to achieve the goal. Their sum communicates the plan's 's overall quality.
Our approach can also be viewed as a learning-based method to integrate mid-level and high-level controllers together, where demonstrations from both are available train-time, only the high- level controller is available at test-time, and the high-level controller can vary. The high-level controller's action specifies subgoal for the mid-level controller. A density model of future trajectories of an expert mid-level controller is learned at train-time, and is amenable to different types of direction as specified by the high-level controller. In this sense, the model is an "apprentice", having learned to imitate mid-level behaviors. In our application, the high-level controller is composed of an A path-planning algorithm and one of library of components that forms goal likelihoods from the waypoints produced by A". Connecting this to related approaches, learning the mid- level controller (Imitative Model) resembles offline IL. whereas inference with an Imitative Model resembles trajectory optimization in MBRL, given goals provided by the high-level controller.
2.2 CONSTRUCTING GOAL LIKELIHOODS
Constraint-based planning to goal sets (hyperparameter--ree)) Consider the setting where we have access a set of desired final states, one of which the agent should achieve. We can model this by applying Dirac-delta distribution on the final state, to ensure it lands in a goal set G C CR.:
p[]|s, Φ) + Rem (G), Jøy (G)= = if ST E G, (G) = if ST E G.
(2)
Sey. (G)'s partial support of ST G C RD constrains sr and introduces no hyperparameterss into p[G]s, [s, Φ). For each choice of G. we have a different way to provide high-level task information to the agent. The simplest choice for G is a finite set of points: (A) Final-State Indicator likelihood. We applied (A) to sequence of waypoints received from a standard A planner (provided by the CARLA simulator), and outperformed all prior dynamic-world CARLA methods known us. We can also consider providing an isfinite number of points. Providing set of line-segment as G yields a (B) Line-Segment Final-State Indicator likelihood, which encourages the final state to land along one of the segments. Finally, consider a (C) Region Final-State Indicator likelihood in which G a polygon (see Figs. and 4). Solving Eq. I with (C) amounts to planning the most expert-like trajectory that ends inside a goal region. Appendix B provides derivations, implementation details, and additional visualizations. We found these methods to work well when & contains "expert-like" final position(s), as the prior strongly penalizes plans ending in non-expert-like positions.
Unconstrained planning to goal sets (typerparameter-bast
Instead of constraining that the final state of the trajectory reach goal, we can use goal likelihood with full support ( RR)). centered at a desired final state. This lets the goal likelihood encourage goals, rather than dictate them. If there is single desired goal (G= (G=(gr) {gr)), }), the (D) Gaussian Final-State likelihood p(g)s, h) <
3

---

Published as a conference paper at ICLR 2020
(ggr sy, ef) treats gr noisy observation of final future state, and encourages the plan to arrive at a final state. We can also plan to K successive states 9 = (gr-Kt1..... gr with a (E) Gaussian State Sequence: p(g]s, Ø) < I ++ (gg: [,, if a program wishes to specify desired end velocity or acceleration when reaching the final state gr (Fig. 2). Alternatively, planner may propose set of states with the intention that the agent should reach any one of them. This is possible by using a (F) Gaussian Final-State Mixture: p(G Φ) < R useful if some of those final states are not reachable with an expert-like plan. Unlike A-C, D-F introduce a hyperparameterr "c". However, they are useful when no states in G correspond to observed expert behavior, they allow the imitation prior be robust to poorly specified goals.
: e!) and
Our model has the additional flexibility to accept arbitrary user-specified costs c at test-time. For example, we may have updated knowledge of new hazards at test-time, such as given map of potholes or predicted cost map. Cost-based knowledge ((s: can be incorporated as an (G) Energy-based likelihood: p(l)s, d) II' -<((ll))) (Todorov, 2007: Levine, 2018). This can be combined with other goal-seeking objectives by simply multiplying the likelihoods together. Examples of combining (energy-based) with F (Gaussian mixture) were shown in Fig. and are shown in Fig. 3. Next, we describe instantiating ((} in CARLA (Dosovitskiy et al., 2017).
Costed planning:
Designing general goal likelihoods can be considered a form of reward engineering if there are no restrictions on the goal likelihoods. This connection is best seen in (G), which has an explicit cost term. One reason why it is easier design goal likelihoods than to design reward functions is that the task of evoking most aspects of goal-driven behavior is already learned by the prior (s]()), which models desirable behavior. This contrast to model-free RL, which entirely relies on the reward design to evoke goal-driven behavior, and in contrast to model-based RL, which heavily relies on the reward design to evoke goal-driven behavior, as its dynamics model learns what is possible, rather than what is desirable. Additionally, it is easy to design goal likelihoods when goals provide significant amount of information that obviates the need to do any manual tuning. The main assumption is that one of the goals in the goal set is reachable within the model's 's time-horizon.
Cloce Tue poRtEMA
Claves piu Fart partiins
ANN ANE IMA
Figure 2: Imitative planning with the Gaussian State Sequence enables fine- grained control of the plans.
Figure 3: Costs can be assigned to "pot- holes" only seen at test-time. The plan- ner prefers routes avoiding potholes.
Figure 4: Goal regions can be coarsely specified to give directions.
2.3 APPLYING DEEP IMITATIVE MODELS TO AUTONOMOUS DRIVING
In our autonomous driving application, we model the agent's 's state at time i as E RD with D D=2; 2; St represents our agent's location on the ground plane. The agent has access to environment perception + (s- X. }}. where is the number of past positions observation of the scene, and À is a oov-dimensional traffic light signal. X could represent either LIDAR camera images (or both), and is the agent's 's observation of the world. In our setting, featurize LIDAR to X = x000xx00x2 with Xij representing a 2-bin histogram of points above and at ground level in 0.5m² 2 cell at position (i. CARLA provides ground-truth s- and A. Their availability is realistic input assumption in perception-based autonomous driving pipelines.
condition on, is a high-dimensionall
Model requirements: A deep imitative model forecasts future expert behavior. must be able to compute (s) E RT'D The ability to compute Vss(s|Ø) enables maddient-based optimization for planning. Rudenko et al. (2019) provide a recent survey on forecasting agent behavior. As many forecasting methods cannot compute trajectory probabilities, we must be judicious in choosing (S ()) model that can compute probabilities R2P2 (Rhinehart et al., 2018), a generative autoregressive flow (Rezende & Mohamed, 2015; Oord et al., 2017). We extend R2P2 to instantiate the deep imitative
4

---

Published as a conference paper at ICLR 2020
Expert examples
Imitative Model
Ronte planner
Path planning
PID Controller
Training phose-
Test pluase
Warpeate
*0ge
Chosen plan
(($1TT0)
Figure 5: Illustration of our method applied to autonomous driving. Our method trains an imitative model from dataset of expert examples. After training, the model repurposed as an imitative planner. At test-time, route planner provides waypoints to the imitative planner, which computes expert-like paths to each goal. The best plan is chosen according to the planning objective and provided to a low-level PID-controller in order to produce steering and throttle actions. This procedure is also described with pseudocode in Appendix A.
model (SS))) R2P2 was previously used to forecast vehicle trajectories: it was not demonstrated
or developed to plan or execute controls. Although we used R2P2, other future-trajectory density estimation techniques could be used designing (()()) is not the primary focus of this work. In
R2P2, qo (S))) is induced by an invertible, differentiable function: S = fo (Z; o): T x RTTx
So warps a latent sample from base distribution Z go =N 1) to S. is trained to maximize 4o (|6) of expert trajectories. So is defined for 1..7 as follows:
S. = f( - (S (: Φ) + 10
where no (S ()) S - S. ma (S
-1 S my (S 1: Φ)
encodes constant-velocity inductive bias. The me E 2 and 0g E x²x2 are computed by expressive neural networks. The resulting trajectory distribution is complex and multimodal (Appendix C.I
depicts samples). Because traffic light state was not included in the of R2P2's "RNN" model, it could not react to traffic lights. We created new model that includes X. It fixed cases where (S 6) exhibited forward- forward-moving preference when the agent already stopped, and improved (S stopping preference at red lights. We used T 40 trajectories at 10Hz (4 seconds), and T = Fig. 12 in Appendix C depicts the architecture of po and ag.
2.4
IMITATIVE DRIVING
We now instantiate a complete autonomous driving framework based on imitative models to study our experiments, seen in Fig. We use three layers of spatial abstraction to plan to a faraway destination, common autonomous vehicle setups: coarse route planning planning within the observable space, and feedback control to follow the planned path (Paden et al., 2016; Schwarting et al., 2018). For instance, route planner based navigation system might output waypoints roughly in the lanes the desired direction of travel, but not accounting for environmental factors such as the positions of other vehicles. This roughly communicates possibilities of where the vehicle could go, but not when or how it could get to them, or any environmental factors like other vehicles. A goal likelihood from Sec. 2.2 is formed from the route and passed to the planner, which generates a state-space plan according to the optimization in Eq. 1. The resulting plan fed to simple PID controller on steering, throttle, and braking. Pseudocode of the driving and inference algorithms are given in Algs I and 2. The PID algorithm is given Appendix
road map. path
conventional GPS-bor based
RELATED WORK
body of previous work has explored offline II (Behavior Cloning - BC) in the CARLA simulator (Li et al., 2018; Liang et al., 2018; Sauer et al., 2018; Codevilla et al., 2018; 2019). These BC approaches condition on goals drawn from a small discrete set directives. Despite BC's theoretical drift shortcomings (Ross et al., 2011), these methods still perform empirically well. These approaches and ours share the same high-level routing algorithm: an A planner on route nodes that generates waypoints. In contrast to approach, these approaches use the waypoints in Waypoint Classifier, which reasons about the map and the geometry the route to classify the waypoints into one of several directives: (Turn left, Turn right, Follow Lane, Go Straight). One of the original motivations for
5

---

Published as a conference paper at ICLR 2020
Algorithm 1 IMITATTVESRRVINNGRRTTTTTE IMITATIVEPLAN, PIDCONTROLLER, qu, J.p, H)
1: @ ENVIRONMENT @) (Initialize the robot)
2: while not at destination do
3: g < (OUTEPLLAN()) (Generate goals from route)
4:
$ f- IMITATIVEPLAN(()) J.P. () (Plan path)
5:
for / = O to // do
6:
PIDCONTROLLER( sy. h, H)
7:
④ <- ENVIRONMENT(1) {Execute control)
8:
end for end while
9:
Algorithm 2 INITATIEELLNNq..DD.....
1: Initialize Z1.r ~
2: while not converged do
3: Z1:T < R1.+VV.. xnr (0g
(Gradient ascent on Eq. 1)
4: end while
5: return S1.T (Z11T)
these type of controls was to enable a human to direct the robot (Codevilla et al., 2018). However, in scenarios where there is no human in the loop (i.e. autonomous driving), we advocate for approaches to make use of the detailed spatial information inherent in these waypoints. Our approach and several others we designed make use of this spatial information. One of these is CIL-States (CILS): whereas the approach in Codevilla et al. (2018) uses images to directly generate controls, CILS uses identical inputs and PID controllers as our method. With respect to prior conditional IL methods, our main approach has more flexibility to handle more complex directives post-training, the ability to learn without goal labels, and the ability to generate interpretable planned and unplanned trajectories. These contrasting capabilities are illustrated in Table 1.
Our approach is also related to MBRL. MBRL can also plan with a predictive model, but its model only represents possible dynamics. The task of evoking expert-like behavior is offloaded to the reward function, which can be difficult and time-consuming craft properly. We know of no MBRL approach previously applied to CARLA, we devised one for comparison. This MBRL approach also uses identical imputs to our method, instead to plan a reachability tree (alle, 2006) over an dynamic obstacle-based reward function. See Appendix D for further details of the MBRL and CILSS methods, which we emphasize use the same inputs as our method.
Several prior works (Tamar et al., 2016; Amos et al., 2018; Srinivas al., 2018) used imitation learning to train policies that contain planning-like modules as part of the model architecture. While work also combines planning and imitation learning, ours captures distribution over possible trajectories, and then plan trajectories at test-time that accomplish a variety of given goals with high probability under this distribution. Our approach is suited to offine-learning settings where interactively collecting data is costly (time-consuning or dangerous). However, there exists online II. approaches that seek to be safe (Menda et al., 2017; Sun et al., 2018; Zhang & Cho, 2017).
4 EXPERIMENTS
We evaluate our method using the CARLA driving simulator (Dosovitskiy et al., 2017). We seek to answer four primary questions: (1) Can we generate interpretable, expert-like plans with offline learning and minimal reward engineering? Neither II. nor MBRL. can do so. It is straightforward to interpret the trajectories by visualizing them on the ground plane; we thus seek to validate whether these plans are expert-like by equating expert-like behavior with high performance on the CARLA benchmark. (2) Can achieve state-of-the-art CARLA performance using resources commonly available in real autonomous vehicle settings? There are several differences between the approaches, discussed in Sec 3 and shown in Tables and 2. Our approach uses the CARLA toolkit's 's resources that are commonly available in real autonomous vehicle settings: waypoint-basd routes (all prior approaches these), LIDAR and traffic-light observations (both are CARLA-

---

Published as a conference paper at ICLR 2020
Table 1: Desirable attributes of each approach. A green check denotes that a method has a desirable attribute, whereas red cross denotes the opposite. A "i" indicates an approach we implemented.
Approach
Flexible New Goals
Trains without goal labels
Outputs Plans
Trains Oflline
Has Expert P.D.F.
CIRL (Liung et 2018)
x
CAI." [Sazer et al., 2018)
MT
2018)
CII." (Cadevilla et al., 2018)
X
X
CILRS (Codevillaa et al., 2019)
CILSt
X
MBRL
Imitative Models (Ours)
Table 2: Algorithmic components of each approach. A "f" indicates an approach we implemented.
Approach
Control Algorithm
Leamiing Algorithm
Goal Geneeaation Algurichm
+- Routing Algurichm
High Dtm. Ubs.
CIRL (Liang al., 2018)
Policy
Behavior Cleming-R.. RI.
Waypwint Classifer
A. Waypsinter
Imagge
CAL" (Saser al. 2018)
PID
Affordance Leuming
Waypoiat Classifier
A. Waypointer
(1. 2018)
Policy
Behavior Cloning
Waypoiur Classifler
A Wayssiater
(Codevilla 2018)
Policy
Behavior Ckeing
Waypoint Clasifier
Waypoiner
Imsuge
cn.RS' (Cadevillaa al., 2019)
Policy
Behavior Clening
Waypwint Clssifeer
A" Waypsinter
Imagee
CILS
PID
Trijectary Regresoor
Waywiit Classifer
A' Waypeinter
(LIDARR,))
MBRL.
Reachubility Tree
State Regressur
Waspwint Selector
AT Waypointer
(LIDAR.A)
Inniative Medels (Ourz)
Imitative Plan+PID
Densiyy Est.
Glail Hicliboods
Waypsiner
(JDARR.))
provided, but only the approaches implemented use it). Furthermore, the two additional methods of comparison we implemented (CILS and MBRL) use the exact same inputs as our algorithm. These reasons justify an overall performance comparison to answer (2): whether we can achieve state-of-the-art performance using commonly available resources. We advocate that other approaches also make use of such resources. (3) How flexible is our approach to new tasks? We investigate (3) by applying each of the goal likelihoods we derived and observing the resulting performance. (4) How robust is our approach to error the provided goals? We do so by injecting two different types of error into the waypoints and observing the resulting performance.
We begin by training (S))) on a dataset of 25 hours of driving we collected in Town01, detailed in Appendix C.2. Following existing protocol, each test episode begins with the vehicle starting in one of a finite set of starting positions provided by the CARLA simulator in Town01 or Town02 maps in one of two settings: static- woold (no other vehicles) or dynamic-world (with other vehicles). We ran the same benchmark 3 times across different random seeds to quantify means and their standard errors. We construct the goal set G for the Final-State Indicator (A) directly from the route output by CARLA's waypointer. B's line segments are formed by connecting the waypoints to form a piecewise linear set of segments. C's regions are created a polygonal goal region around the segments of (B). Each represents an increasing level of coarseness of direction. Coarser directions are easier to specify when there ambiguity in positions (both the position of the vehicle and the position of the goals). Further details are discussed in Appendix B.3. Visualizations of (C) are shown in Figures 6 and 7. Visualizations of (A) and (B) are shown in Figures 8 and 9. We use three metrics: (a) success rate in driving to the destination without any collisions (which all prior work reports); (b) red-light violations; and (c) proportion of time spent driving in the wrong lane and off road. With the exception of metric (a). lower numbers are better.
Results: Towards questions (1) and (3) (expert-like plans and ficxibility), we apply our approach with variety of goal likelihoods to the CARLA simulator. Towards question (2), we compare methods against CILS, MBRL, and prior work. These results are shown in Table 3. The metrics for the methods we did not implement are from the aggregation reported in Codevilla et al. (2019). We observe our method to outperform all other approaches in all settings: static world, dynamic world, training conditions, and test conditions. We observe the Goal Indicator methods are able to perform well, despite having no hyperparameters to lune. We found that we could further improve our approach's 's performance if we use the light state to define different goal sets, which defines a "smart" waypointer. The settings where we use this are suffixed with "S." in the Tables. We observed the planner prefers closer goals when obstructed, when the vehicle was already stopped, and when a red light was detected; we observed the planner prefers farther goals when unobstructed and when green lights or no lights were observed. Examples of these and other interesting behaviors are best seen in the videos on the website pectps://sitee .oosgg..o.....ooooo www.cccoo////eecccc /mmi tative- tmitalieemmddes).. These behaviors follow from the method leveraging q(S) 6)'s internalization of aspects of expert behavior in order to reproduce them in new situations. Altogether, these results provide affirmative answers to questions (1) and (2). Towards question (3), these results show that our approach is flexible to different directions defined by these goal likelihoods.
7

---

Published as a conference paper at ICLR 2020
Table We evaluate different autonomous driving methods on CARLA's Dynamic Navigation task. A "t" indicates methods we have implemented (each observes the same waypoints and LIDAR as input). **" indicates results reported in Codevilla al. (2019). A "_" indicates an unreported statistic. A "!" indicates optimistic estimate in transferring result from the static setting to the dynamic setting. "S." denotes a "smart" waypointer reactive to light state, detailed in Appendix B.2. Results accompanied by standard errors are computed with N 3 trials across environment seeds.
Touno: (raming conditions)
Tount2 (est conditios)
Dysamic
Secssna
Rax Light
Wony bece
Od rsaal]
Suaszed
Real Lacle
Winny banc
rend.
inng (LI 2018)
91%
55%
CII-MC.sdeillle ct 2019)
83%
838--
38%
$261
CILS, Waygsint Inguz!
0,0%
0.30%
L111%
11.70%
Waypoint Iupus"
64%
72%
11.1%
20.65
13.3
moilld, Region Fina-Si, Indiiattr
965-11
0,000
0.11%aaon
188
049%-113
2.00%
Our wetlod. Regton Pinak-rr Indicator
9256-23
18%
S1%615
54.756115
1:322 ea
Our wachod, Live Segneent Fanal St. Inlicator
9158-11 1.
0.0L55 L000
5875435
36.2% 144
0.525%ua43
0.1855
Our wathod, Final Staee Indicator"
925
26%
0.05%
0.012%6
$68
35%
0.13%
0.28%
Our recheot Gouzsun Final-ii. Mic!!
9358
6.3%
0.04%
0.005%
1009
1268
0.11%
Our reclest Gaveson Flai--tt Mic. S."
100%
1.7%
6.03%
0,005%
928
0.0%
0.05
0.159
Tosn01 (trining conditions)
Torno2 (eest condicions)
Static Now. Method
Success?
Ran Red Light
Wrong lane)
OH road)
Success1
Ran Red Light.
Wrong lane)
read)
CIRI." (1 (nnn et al., 2018)
91%
68%
CAL." (Sancr et al., 2018)
92%
688%
MT" (Li et al., 2018)
81%
78%
CIL. (Codevillaa
2018)
8656
83%
44%
K25
CII, CIT.SS (CCdeeillllllllle et 2019)
95%
27%
9096
64%
CIL.S, Waypaint Inpar!
28%
10.25
0.0%
1,69%
168828
MBKL, Woypoint lupus!
78%
14.3%5
194%
735
0.75%
Our mothod, Final State Indicator
100%
48%
0.0595
0.002%
100%
52%
0.10%
0.13%
Our method. Gourzian Final St. Mismrre
96%
0,83%
0.015
0.08%
95%
0.0%
0,83%
0,1416
Our method Gourrian Final- St. Mis. S.7
95%
0,0%
0.04%
0.07%
92%
0.0%
0.18%
0.77
Figure 6: Planning with the Region Final State Indicator yields plans that end inside the region. The orange polygon indicates the region. The red circles indicate the chosen plan.
Even with wider goal region than Fig. 6, the vehicle remains in its lane. Despite their coarseness, these wide goal regions still provide useful guidance to the vehicle.
Figure 7:
4.1 ROBUSTNESS TO ERRORS IN GOAL-SPRCIIICATION
Towards questions (3) (flexibility) and (4) (noise-robustness), we analyze the performance of our method when the path planner is heavily degraded, to understand its stability and reliability. We use the Gaussian Final-State Mixture goal likelihood.
Navigating with high-variance waypoints. As a test of our model's capability to stay in the distribution of demonstrated behavior, we designed "decoy waypoints" experiment, in which half of the waypoints are highly perturbed versions of the other half, serving as distractions for our Gaussian Final-State Mixture imitative planner. We observed surprising robustness to decoy waypoints. Examples of this robustness shown in Fig. 10. In Table 4, report the success rate and the mean number of planning rounds for failed episodes in the "1/ distractors" row. These numbers indicate method causing a catastrophic failure, and often can execute the hundreds necessary to achieve the goal. See Appendix E for details.
execute dozens of planning rounds without decoy waypoints
Navigating with waypoints on the wrong side of the road. We also designed an experiment to test our method under systemic bias in the route planner. Our method is provided waypoints on the
8

---

Published as a conference paper at ICLR 2020
Figure 8: Planning with the Final State Indicator yields plans that end at one the provided loca- tions. Orange diamonds indicate the locations in the goal set. Red circles indicate the chosen plan.
Figure 9: Planning with the Line Segment Final State Indicator yields plans that end along a seg- ment. Orange diamonds indicate line segment endpoints. Red circles indicate the chosen plan.
man Kas peolives
Caamm plan poolicns
Dowes dass padkess Basley
Figure 10: Tolerating bad goals. The planner prefers goals in the distribution of expert behavior (on the road at a reasonable distance). Left: Planning with 1/2 decoy goals. Right: Planning with all goals on the wrong side of the road.
Figure 11: Test- time plans steering around potholes.
wrong side of the road (in CARLA, the left side), and tasked with following the directions of these waypoints while staying on the correct side of the road (the right side). In order for the value of ((s()) to outweigh the influence of these waypoints, we increased the hyperparameter. We found our method to still be very effective at navigating, and report results in Table 4. We also investigated providing very coarse 8-meter wide regions to the Region Final- State likelihood; these always include space in the wrong lane and off-road (Fig. 7 in Appendix ? provides visualization). Nonetheless, Town01 Dynamic, this approach still achieved an overall success rate 48%. Taken together towards question (4), results indicate that method is fairly robust to errors in goal-specification.
4.2 PRODUCING UNOBSERVED BEHAVIORS AVOID NOVEL OBSTACLES
Table 4: Robustness to waypoint noise and test-time pothole adaptation. Our method is robust to waypoints on the wrong side of the road and fairly robust to decoy waypoints. Our method is flexible enough to safely produce behavior not demonstrated (pothole avoidance) by incorporating test-time cost. Ten episodes are collected in each Town.
Tosn01 (training conditions)
Tonn02 conditions)
Waypointer
Extra Cost
Success
Wrong
Pothales
Success
Wrong lane
Potholes
Noiscless waypointer Waypaints wrong lane 1/2 way peypoints distracting
100%
0.00%%
177/230
100%
0.41%
82/154
100%
0.34%
70%
3.16%
70%
50%
waypointer
35/154
To further investigate our model's flexibility to test-time objectives (question 3), we designed pothole avoidance experiment. We simulated potholes in the environment by randomly inserting them in the cost map near waypoints. We ran our method with test-time-only cost map of the simulated potholes by combining goal likelihoods (F) and (G), and compared to our method that did not incorporate the cost map (using (F) only, and thus had no incentive to avoid potholes). We recorded the number of collisions with potholes. In Table 4, method with cost incorporated avoided most potholes while avoiding collisions with the environment. To do so, drove closer to the centerline, and occasionally entered the opposite lane. Our model internalized obstacle avoidance by

---

Published as a conference paper at ICLR 2020
staying on the road and demonstrated its flexibility to obstacles not observed during training. Fig. 11 shows an example of this behavior. See Appendix for details of the pothole generation.
5 DISCUSSION
We proposed "Imitative Models" to combine the benefits of IL and MBRL. Imitative Models are probabilistic predictive models able plan interpretable expert-like trajectories to achieve new goals. Inference with an Imitative Model resembles trajectory optimization in MBRL, enabling it to both incorporate new goals and plan them test-time, which IL. cannot. Learning an Imitative Model resembles offline IL, enabling it to circumvent the difficult reward-engineering and costly online data collection necessities of MBRL. We derived families of flexible goal objectives and showed model can successfully incorporate them without additional training. Our method substantially outperformed six IL approaches and an MBRL approach in a dynamic simulated autonomous driving task. We showed our approach is robust poorly specified goals, such as goals the wrong side of the road. We believe our method is broadly applicable in settings where expert demonstrations available, flexibility to situations is demanded, and safety is paramount. Future work could investigate methods to handle both observation noise and out-of-distribution observations to enhance the applicability robust real systems more general planning, future work could extend our approach to explicitly reason about all agents in the environment in order inform closed-loop -lop plan for the controlled agent.
expand this issue in Appendix E. Finally, facilitate
ACKNOWLEDGEMENTS
This research was supported by ONR N000141712623, DARPA Assured Autonomy, ARL DCIST CRA W911NF-11-2-0111, Google, NVIDIA, and Amazon.
REFERENCES
Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and Zico Kolter. Differ- entiable MPC for end-to-end planning and control. In Neural Information Processing Systems (NeurIPS), 2018.
Felipe Codevilla, Matthias Miller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. International Conference on Roboties and Automation (ICRA), pp. 1-9. IEEE, 2018
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. arXiv preprint 1904.0880 2019.
Marc Deisenroth and Carl E Rasmussen. PILCO: A model-based and data-efficient approach to
policy search. In International Conference on Machine Learning (ICML), pp. 465-472, 2011.
Alexey Dosovitskiy and Vladlen Koltun. Learning act by predicting the future. arXiv preprint arXiv:1611.0117,, 2016.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Conference on Robot Learning (CoRL), pp. 1-16, 2017.
Leonid Kuvayev and Richard S. Sutton. Model-based reinforcement learning with approximate, learned model. In Yale Workshop on Adaptive and Learning Systems, pp. 101-105, 1996.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (NeurIPS), pp. 1400264433 2017.
Steven M Lalalle. Planning algorithms. Cambridge University Press, 2006.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 1rX5::1805.n (00..000550090 2018
10

---

Published as a conference paper at ICLR 2020
Zhihao Li, Toshiyuki Motoyoshi, Kazuma Sasaki, Tetsuya Ogata, and Shigeki Sugano. Rethinking self-driving: Multi-task knowledge for better generalization and accident explanation ability. arXiv preprint
100, 2018.
Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. CIRL: Controllable imitative reinforcement
learning for vision-based self-driving. arXiv preprint arXiv:/807.03776 2018.
Kunal Menda, Katherine Trrggs-CCambell.. and Mykel J Kochenderfer. DropoutDAgger. A Bayesian approach safe imitation learning. arXiv preprint arXiv:1709.06166, 2017.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel WaveNet: Fast high-fidelity speech synthesis. arXiv preprint arXiv: 2017.
Brian Paden, Michal Cáp, Sze Zheng Yong, Dmitry Yershov and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. Transactions on Intelligens Vehicles, 1(1):33-55, 2016.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), pp. 1530-1538, 2015.
Nicholas Rhinehart, Kris M. Kitani, and Paul Vernaza. R2P2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In European Conference on Computer Vision (ECCV), September 2018.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS), pp. 627-635, 2011.
Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitami, Dariu Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. arXiv preprint arXiv: 1905.060111111 13, 2019.
Axel Sauer, Nikolay Savinov, and Andreas Geiger. Conditional affordance learning for driving in urban environments. arXiv preprint arXiv: 1806. 1806.0649,, 2018.
Wilko Schwarting, Javier Alonso-Mora, and Daniela Rus. Planning and decision-making for au- tonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 1:187-210, 2018.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. In International Conference Machine Learning (ICML), pp. 4739 2018.
Liting Sun, Cheng Peng, Wei Zhan, and Masayoshi Tomizuka. A fast integrated planning and control framework for autonomous driving via imitation learning. In Dynamic Systems and Control Conference. American Society of Mechanical Engineers, 2018.
Aviv Tamar, Yi Wu. Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks, In Neural Information Processing Systems (NeurIPS), pp. 2154-2162, 2016.
Emanuel Todorov. Linearly-solvable Markov decision problems. In Neural Information Processing Systems (NeurIPS), pp. 1369-1376, 2007.
Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simulated driving. In Association for the Advancement of Artificial Intelligence (AAAI), pp. 2891-28977 2017.
11

---

Published as a conference paper at ICLR 2020
A ALGORITHMS
In Algorithm 1, we provided pseudocode for receding-horizon control via our imitative model. In Algorithm 2 provided pesudocode that describes how plan in the latent space of the trajectory. In Algorithm 3, we detail the greed-based throttle and position-based steering PID controllers.
Algorithm 3 PIDCONTROLLER = {sp, 5 s h, H; K Kp)
1: < T H h (Compute the index of the target position]
2: Sprocess-speed t- (50,2) 1.1)) [Compute the current forward speed from the observations]
3: Syetpoint-ooiiiion E 1, [Retrieve the target position x-coordinate from the plan)
4: Suelpoint- speed f- poiinuss (Compute the forward target speed]
5: éa & Sserpoint-speed Spracess-speed (Compute the forward speed error)
6: Wg +- K (Compute the accelerator control with a nonzero proportional term)
7: throttle (e > 0).u (e 0) (Use the control as throttle if the speed error is positive)
8: brake +- (e > 0+ I 0) u (Use the control as brake if the speed error is negative)
9: Oprocess f arctan - 5-1.y S0.z 5_1,a) [Compute current heading)
10: Osetpponnt f- arctan .y So.y: is Sq.a. (Compute target forward heading)
11: Ca Ossetpoint Oppncess (Compute the heading error)
12: steering f- Kp co. [Compute the steering with a nonzero proportional term
13: u. + [throttle, steering, brake]
return
A.1 LATENT PLAN OPTIMIZATION
Since S1:T = f(ZIT) our implementation, and f is differentiable, we can perform gradient descent of the same objective in terms of Z1:r, shown in Algorithm 2.Since is trained with Z1:r ~ (0, I), the latent space is likelier to be better numerically conditioned than the space of S1.T, although we did not compare the two approaches formally. We implemented the following optimizations to improve this procedure's output and practical run time. 1) We started with N 120 different z initializations, optimized them in batch, and returned the highest-scoring value across the entire optimization. 2) We observed the resulting planning procedure to usually converge quickly, so instead of specifying convergence threshold, we simply ran the optimization for a small number of steps, M = 10, and found that we obtained good performance. Better performance could be obtained by performing larger number of steps.
B GOAL DETAILS
B. OPTIMIZING GOAL LIKELIHOODS WITH SET CONSTRAINTS
We now derive an approach optimize our main objective with set constraints. Although we could apply a constrained optimizer, we find that we are able to exploit properties of the model and constraints to derive differentiable objectives that enable approximate optimization of the cor- responding closed optimization problems. These enable to the same straightforward gradient-descent- optimization approach described in Algorithm
Shorthand notation: In this section we omit dependencies on é for brevity, and use short hand 14: 14 (81:--1))) and Σ, Eg (8) For example, (( = M (s:; Mt, Et).
Let us begin by defining useful delta function:
11 if
6 (G)
(4)
0 if ST GG
12

---

Published as a conference paper at ICLR 2020
which serves as goal likelihood when using goal with set constraints: p(G/51:7) +- (G). We now derive the corresponding maximum a posteriori optimization problem:
s iit
arg max p()):r(9)
81.7 ER
p( S1:T) ((S:T)) p (g)
arg max
8 ER
arg max
(
S1:7)
S1:77
goal likelihood
imitation prior
arg max
Ösr (G)
(1.T)
$1.T
set constraint
imitation prior
(((:T))
ST E G
arg max
(0
if ST &f G (81:T)
$1.7 ER
arg max $1:7-1 (R2(T-1) eG
T-1
arg arg max g(sr ((st/s1:1)) (1 sreg
II (s
t=1
T-1
arg max max W(st: pT, Ex) SEP- ERT-IER Er
II
N(s 14, Ex).
(5)
By exploiting the fact that ((5r) 11) = N (ST: pr, Er), we can derive closed-form solutions for
arg max N (ST: pT, (TT Σt) SrEG
(6)
when G has special structure, which enables apply gradient descent solve this constrained- optimization problem (examples below). With a closed form solution to equation 6, we can easily compute equation 5 using urconstrainedd-pptiiizatoo follows:
T-1
S 1
aig max arg max
g(sT
II g(s)
(R2(T-1) By EG
T
11T
arg max SETTIEERATTD
Il (5:
(8)
unconstrained optimization
objective function of 61:2-1
Note that equation 8 only helps solve equation 5 if equation 6 has closed-form solution. We detail example of goal-sets with such dosed-form solutions in the following subsections.
B.1.1 POINT GOAL-SET
The solution equation 6 in the case of single desired goal y E RP is simply:
Gpuint
- {gr).
(9)
arg N (st: Er) Epun
point
gr-
(10)
More generally, multiple point goals help define optional end points for planning: where the agent only need reach one valid end point (see Fig. 8 for examples), formulated as:
G Epoints
(11)
Si points
arg max (gt: Er)
(12)
g GGpin
13

---

Published as a conference paper at ICLR 2020
B.1.2 LIGE-SEGMENT GOAL-SET
We form goal-set as a finite-length line segment, connecting point RD to point I
Itine u) a a u. (b - a), u E R,
(13)
Ga-b = [u) E [0, 1]].
(14)
line segment
The solution equation in the case of sinemetgment goals is:
ST line-segment
arg max G Innseenee
N (sT; prr Er)
(15)
1, max (0, (b (b-a) a) MT (UT - =
)) (b. a). (b a) ET (b a)
a min
(16)
Proof:
To solve equation 15 is to find which point along the line ginne maximizes W (T, Σr) subject the constraint 0 S < 1:
ux arg max (glino (u): pT, Er)) w
arg min (Unin(u) - UT) T "TT"" (ytine (u) - pT)
(17)
we(0,1]
Cu(u)
Since f, convex, the optimal value u² value closest to the unconstrained arg max of L. (u).
subject to 0 S 1:
u = arg max (u),
(18)
WEB
u² arg min Cu(w)
we(0,1]
min (1, max (0, ux)).
(19)
We now solve for uz:
dL(u)
d (giine (u) MT) " (Oliee (u) (TT)
U = u::0
du
du
2 d(ginn (u) - HTT)
DTT (gline (u) WT)
du
2 d(a (b -
Σg (a u u.- (b-a) a) -er)
du
2. (b- a) T 'n (a (atto u (b - a) - -1T),
(b a) ΣT (NT -a)
ug (b a) Eg (b a
(20)
which gives us:
ST. line segment
Gline (u")
a + u² (b a)
= a + min (1, max (0, 2x)) (b a)
+ min 1, max 0, (b a) Ex (MT - a)
(b a) Mi (b- a) )) (b a).
(21)
B.1.3 MULTIPLE-LINE-SSONE GOAL-SET:
More generally, to follow. By defining a path that connects points (Po. Pi.... PN). we can evaluate C for each G GP:pPi+1
combine multiple line-segments to form piecewise linear "paths" wish
segment select the optimal segment i arg max, Li and the segment i* solution u² to compute sy Examples shown in Fig. 9.
14

---

Published as a conference paper at ICLR 2020
B.1.4 POLYGON GOAL-SET
Instead of route or path, user (or program) may wish to provide a general region the agent should go to, and state within that region being equally valid. Polygon regions (including both boundary and interior) offer closed form solution to equation 6 and are simple to specify. A polygon can be specified by an ordered sequence of vertices (Po. (Po-PI,... Px) E I XNxx Edges are then defined as the sequence of line-segments between successive vertices (and final edge between first and last vertex): ((Po, PI)..., (PN PNN), (PN, Po)). Examples shown in Fig. 6 and 7.
Solving equation with polygon has two cases: depending whether pr is inside the polygon, or outside. If ur lies inside the polygon, then the optimal value for s that maximizes (s i ur, Er) simply μr: the mode of the Gaussian distribution. Otherwise, if μr lies outside the polygon, then the optimal value s will lie on one of the polygon's 's edges, solved using B.1.3.
B.2 WAYPOINTER DETAILS
The waypointer uses the CARLA planner's provided route to generate waypoints. In the constrained- based planning goal likelihoods, we use this route to generate waypoints without interpolating between them. In the relaxed goal likelihoods, we interpolate this route to every 2 meters, and use the first 20 waypoints. As mentioned the main text, one variant of our approach uses a "smart" waypointer. This waypointer simply removes nearby waypoints closer than 5 meters from the vehicle when a green light is observed in the measurements provided by CARLA, to encourage the agent to move forward, and removes far waypoints beyond 5 meters from the vehicle when red light is observed in the measurements provided by CARLA. Note that the performance differences between method without the smart waypointer and method with the smart waypointer small: the only signal in the metrics is that the smart waypointer improves the vehicle's ability to stop for red lights, however, is quite adept at doing so without the smart waypointer.
B.3 CONSTRUCTING GOAL SETS
Given the in-lane waypoints generated by CARLA's route planner, we use these to create Point goal sets, Line-Segment goal sets, and Polygon Goal-Sets, which respectively correspond to the (A) Final-State Indicator, (B) Line-Segment Final-State Indicator, and (C) Final-State Region Indicator described in Section 2.2. For (A), we simply feed the waypoints directly into the Final-State Indicator, which results in constrained optimization the vehicle's 's current position in the goal set, in order to allow it to stop. The gradient-descent based optimization is then formed from combining Eq. 8 with Eq. 12. The gradient the nearest goal of plan encourage the optimization to move the plan closer to that goal. We used K 10. We applied the same procedure to generate the goal set for the (B) Line Segment indicator, as the waypoints returned by the planner are ordered. Finally, for the (C) Final-State Region Indicator (polygon), we used the ordered waypoints as the "skeleton" of polygon that surrounds. It was created by adding two vertices for each point v, in the skeleton at distance 1 meter from Vt perpendicular to the segment connecting the surrounding points (V,-1, V(+1). This resulted in a goal set G polygon i Gline segment as it surrounds the line segments. The (F) Gaussian Final-State Mixture goal set was constructed in the same way as (A), and also used when the pothole costs were added.
ensure that ST E G
{ k=1- We also included
the final state of the partially-optimin
For the methods we implemented, the task is to drive the furthest road location from the vehicle's initial position. Note that this protocol more difficult than the one used in prior work Codevilla et al. (2018); Liang et al. (2018); Sauer et al. (2018); Li et al. (201 8); Codevilla et al. (2019), which has no distance guarantees between start positions and goals, and often results in shorter paths.
C ARCHITECTURE AND TRAINING DETAILS
The architecture of ((S))) shown in Table 5.
C.I PRIOR VISUALIZATION AND STATISTICS
We show examples of the priors multimodality in Fig. 13
15

---

Published as a conference paper at ICLR 2020
High Dicseusiona Obsrralkes
r
Latent Random Variailes
Z
Z
Output Stutes
Poet Trajectory
IRNN
M1: 01
22,
pe.: OT
200
a
Interpolate
rish)
RNN
Pt
200
200
Traffe Light Statn
2
conv
CoNV
Figure 12: Architecture of mg and de, which parameterize qo (S|@= 8-:0: })). Inputs: LIDAR X. past-states S-7:0: light-state X, and latent noise ZT.-- Output: trajectory SITT Details in Appendix C.
Table 5: Detailed Architecture that implements S1:7 = f(21T,0)) Typically, T 40, D 2,H = W 200.
Component
Inpur (dimensinnaliry)
Layer or Operation
Ourput [dimensionality]
Details
Stutic fenturizattio of conteat:
ex.e
MapFeat
X H. W.
2D Convolution
2x
H. WC
3 x stride I, ReLu
MapPeat
LS1, W.
2D Comvoltionn
H. W.
3 x scride ReLa,, (2.....)
MapFeat
"x (.. W,
2D Convolution
W.
strde ReLa
PaiRNN
8 [F L.D)
[32]
actoss
Dymomic generation via loop: for (0. N.....I-11
MapFieat
4: [?]]
Interpolate
F(s, [8]
Differentiable interpolation
JointFeet
7o.50. a.
[D +00++32--0 32
Concatenote
FetureRNN
P:
1]
A: [50]
PutureMLP
p: (50)
Affine (FC)
Ppi 2200
Tash activation
PetureMPP
"p/2000
Affine (FC) expeats E2 Rrssspooe
mh, [D], & [D, D
Ideniity activation
MatrixExp
& [D.D
[D.D)
Differestiable Matrix Exponential (2018)
VerletStep
Br.8e-1. 0..
25 9i-1 mig + 0.30
Sie: [D]
C.I.1 STATISTICS OF PRIOR AND GOAL LIKELIHOODS
Following are the values of the planning criterion on = 8 10 rounds from applying the "Gaussian Final-State Mixture" Town01 Dynamic. Mean of log q(s (0) 104. Mean of log p(G ,)) = -4. This illustrates that while the prior's value mostly dominates the values of the final plans, the Gaussian Final-State Goal Mixture likelihood has a moderate amount of influence on the value of the final plan.
C.2 DATASET
Before training (S))), we ran CARLA's expert in the dynamic world setting of TownO1 to collect dataset of examples. We have prepared the dataset of collected data for public release upon publication. We ran the autopilot in Town01 for over 900 episodes of 100 seconds each in the presence of 100 other vehicles, and recorded the trajectory of every vehicle and the autopilot's LIDAR observation. We randomized episodes to either train, validation, or test sets. We created sets of 60,701 train, 7586 validation, and 7567 test scenes, each with 2 seconds of past and 4 seconds of future position information at 10Hz. The dataset also includes 100 episodes obtained by following the same procedure in Town02.
D BASELINE DETAILS
D.1 CONDITIONAL IMITATION LEARNING OF STATES (CILS):
We designed a conditional imitation learning baseline that predicts the setpoint for the PID-controller. Each receives the same scene observations (LIDAR) and is trained with the same set of trajectories as main method. It uses nearly the same architecture that of the original CIL, except it outputs setpoints instead of controls, and also observes the traffic light information. We found it very effective for stable control straightaways. When the model encounters corners, however, prediction is more
16

---

Published as a conference paper at ICLR 2020
Figure 13: Left: Samples from the prior, (S))), Φ). go left right. Right: Samples go forward right.
diffieult, in order to successfully avoid the curbs, the model must implicitly plan safe path. We found that using the traffic light information allowed it to stop more frequently.
D.2 MODEL-BASBR REINFORCEMENT LEARNING:
Static-world To compare against a purely model- based reinforcement learning algorithm, we propose model-based reinforcement learning baseline. This baseline first learns a forwards dynamics model 8141 = (St- a₂) given observed expert data (a, are recorded vehicle actions). We use an MLP with two hidden layers, each 100 units. Note that forwards dynamics model does not imitate the expert preferred actions, but only models what is physically possible. Together with the same LIDAR map X our method uses to locate obstacles, this baseline uses its dynamics model plan reachability tree Valle (2006) through the free-space to the waypoint while avoiding obstacles. The planner opts for the lowest-cost path that ends near the goal C(s1:rigT) = ||st grlle + > c(s;), where cost of a position is determined by c(sr) = 1.51(s; < meters from any obstacle) + 0.751(1 <= < 2 meters from any obstacle) + šü.
We plan forwards over 20 time steps using a breadth-first search over CARLA steering angle (-0.3, 0.1, O., 0.3), noting valid steering angles are normalized to [-1, with constant throttle at 0.5, noting the valid throttle range is [0, 1]. Our search expands each state node by the available actions and retains the 50 closest nodes to the waypoint. The planned trajectory efficiently reaches the waypoint, and can successfully plan around perceived obstacles to avoid getting stuck. To convert the LIDAR images into obstacle maps, we expanded all obstacles by the approximate radius of the car, 1.5 meters.
Dynamic-world We use the same setup as the Static-MBRL method, except we add discrete temporal dimension to the search space (one R² spatial dimension per T time steps into the future). All static obstacles remain static, however all LIDAR points that were known collide with a vehicle are now removed: and replaced at every time step using a constant velocity model of that vehicle. We found that the main failure mode was due both to inaceuracy in constant velocity prediction as well as the model's inability to perceive lanes in the LIDAR. The vehicle would sometimes wander into the opposing traffic's lane, having failed to anticipate an oncoming vehicle blocking its path.
E ROBUSTNESS
E. DECOY WAYPOINTS EXPERIMENTS
In the decoy waypoints experiment, the perturbation distribution is AN(0, a 88m): each waypoint is perturbed with standard deviation of 8 meters. One failure mode of this approach is when decoy waypoints lie on a valid oof-route path at intersections, which temporarily confuses the planner about the best route. Additional visualizations shown in Fig. 14.
17

---

Published as a conference paper at ICLR 2020
WALE IEAA
ANN MAL IMA
ANNI
ANN MAM TH
Clome: 288 pollies
Figure 14: Tolerating bad waypoints. The planner prefers waypoints in the distribution of expert behavior (on the road at reasonable distance). Columns 1,2: Planning with 1/2 decoy waypoints. Columns 3,4: Planning with all waypoints on the wrong side of the road.
E.2 PLAN RELIABILITY ESTIMATION
Besides using our model to make best-effort attempt to reach user-specified goal, the fact that our model produces explicit likelihoods can also be leveraged to test the reliability of a plan by evaluating whether reaching particular waypoints will result in human-like behavior or not. This capability can be quite important for real-world safety-critical applications, such as autonomous driving, and can be used to build a degree of fault tolerance into the system. We designed a classification experiment to evaluate how well our model can recognize safe and unsafe plans. We planned our model to known good waypoints (where the expert actually went) and known bad waypoints (off-road) on 1650 held-out test scenes. We used the planning criterion to classify these as good and bad plans and found that we can detect these bad plans with 97.5% recall and 90.2% precision. This result indicates imitative models could be effective in estimating the reliability of plans.
We determined a threshold on the planning criterion by single-goal planning to the expert's ' final location offline validation data and setting it the criterion's mean minus one stddev. Although a more intelligent calibration could be performed by analyzing the information retrieval statistics on the offline validation, we found this simple calibration to yield reasonably good performance. We used 1650 test scenes to perform classification of plans to three different types of waypoints 1) where the expert actually arrived time T (89.4% reliable), 2) waypoints 20m ahead along the waypointer- prrovidde--roiided route. which are often near where the expert arrives (73.8% reliable) 3) the same waypoints from 2). shifted 2.5m off of the road (2.5% reliable). This shows that our learned model exhibits strong preference for valid waypoints. Therefore, waypointer that provides expert waypoints via 1) half of the time, and slightly outtof-distributtion waypoints via 3) the other half, an "unreliable" plan classifier achieves 97.5% recall and 90.2% precision.
E.3 OUT-OF-DISTRIBUTION ROBUSTNESS
The existence of both (1) observation noise and (2) woteaaainnoooooo-ssstiiiiiieeeee observations is an important practical issue for autonomous vehicles. Although our current method only conditions on our current observation, several extensions could help mitigate the negative effects of both (1) and (2). For (1), Bayesian filtering formulation is arguably most ideal, to better estimate (and track) the location of static and dynamic obstacles under noise. However, such high-dimensional filtering are often intractable, and might necessitate approximate Bayesian deep learning techniques, RNNs, or frame stacking, to benefit from multiple observations. Addressing (2) would ideally be done by placing a prior over our neural network weights, to derive some measure confidence in our density estimation of how expert each plan is, such that unfamiliar scenes generate large uncertainty on our density estimate that we could detect, and react cautiously (pessimistically) to. One way very different is to adopt ensembling approach
to address the situation if the distributions Lakshminarayanan et al. (001)) in order for the method to determine when the inputs are out of distribution the ensemble will usually have higher variance (i.e. disagree) when each element of
18

---

Published as a conference paper at ICLR 2020
the ensemble is provided with out-of-distribution input. For instance, this variance could be used as a penalization in the planning criterion.
E.4 TRAFFIC-LIGHT NOISE
As discussed, model assumes access the traffic-light state provided by the simulator, which we call X. However, access to this state would be noisy in practice, because it relies on a sensor- based (usually mmaee-bssed detection and classification module.
We performed an experiment to assess robustness to noise in A: we simulated noise in À by "flipping" the light state with 20% probability, corresponding to a light state detector that has 80% accuracy on average. "Flipping" means that if the light is green, then changing to indicate red, and if the light red, then changing À to indicate green. We performed this following the experimental method of "Region Final-St. Indicator S." in dynamic Town02, and ran it with three separate seeds. The means and their standard errors are reported in Table 6. The conclusion we draw is that the approach can still achieve success most of the time, although it tends to violate red-lights more often. Qualitatively, observed the resulting behavior near intersections to sometimes be "jerky". with the model alternating between stopping and non-stopping plans. We hypothesize that the model itself could be made more robust if the noise in was also present in the training data.
Table 6:
We evaluate the effect of noise in the traffic-light state (A) on CARLA's Dynamic Navigation task. Noise in the light state predictably degrades overall and red-light performance, but not the point of preventing the method from operating at all.
Town02 Dynamic Navigation Method
Success
Ran Red Light
Wrong lane
Off road
Region Final-St. Indicator S. (original)
88% 3.3
2.57%
0.04
0.49% + 0.32
2.6% + 1.06
Region Final-Si. Indicator (noisy X)
76% 5.0
34.8% 1 2.4
0.15% + 0.04
1.79% + 0.34
F POTHOLE EXPERIMENT DETAILS
We simulated potholes the environment by randomly inserting them in the cost map near each waypoint with offsets distributed (x=1|-15m, 0ml, Σ = =,, 0.01])), (i.e. mean centered on the right side of the lane 15m before each waypoint). We inserted pixels of root cost -1e in the cost map at single sample of each N₁, binary-dilated the cost map by 1/3 of the lane-width (spreading the cost neighboring pixels), and then blurred the cost map by convolving with a normalized truncated Gaussian filter of and truncation width 1.
G BASELINE VISUALIZATIONS
See Fig, 15 for a visualization of our baseline methods.
H HYPERPARAMETERS
In order to tune the hyperparameter of the unconstrained likelihoods, undertook the following binary-search procedure. When the prior frequently overwhelmed the posterior, we set e +- 0.2e, to yield tighter covariances, and thus more penalty for failing satisfy the goals. When the posterior frequently overwhelmed the prior, we set + 5e, to yield looser covariances, and thus less penalty for failing to satisfy the goals. We executed this process three times: once for the "Gaussian Final-State Mixture" experiments (Section 4), once for the "Noise Robustness" Experiments (Section 4.1), and once for the pothole-planning experiments (Section 4.2). Note that the Constrained-Goal Likelihoods introduced no hyperparameters to tune.
19

---

Published as a conference paper at ICLR 2020
Figure 15: Baseline methods compare against. The red crosses indicate the past 10 positions of the agent. Left: Imitation Learning baseline: the green cross indicates the provided goal, and the yellow plus indicates the predicted setpoint for the controller. Right: Model-based RL baseline: the green regions indicate the model's 's predicted reachability, the red regions are post-processed LIDAR used create its obstacle map.
20


Reviews:
[{"review_id": "r1xK6GH5YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem: expert-like probabilistic online motion planning to reach arbitrary goals without reward shaping thanks to off-line learning from expert demonstrations;\n- contributions: 1) an imitative planning procedure via gradient-based log-likelihood maximization leveraging \"imitative models\" q(future states | features), 2) multiple proposals to define flexible goals in this probabilistic framework, 3) a complete implementation for end-to-end navigation in CARLA, 4) an extensive experimental evaluation showcasing the performance, flexibility, interpretability, and robustness of the proposed approach w.r.t. the previous state of the art and several Imitation Learning (IL) and Model-Based Reinforcement Learning (MBRL) baselines.\n\nRecommendation: weak accept (leaning towards strong accept)\n\nKey reason 1: principled probabilistic framework bringing the best of both IL and MBRL worlds.\n- this planning as inference method is very succinctly and elegantly described in the paper with enough details in appendix (+ code) to suggest a high chance of reproducibility;\n- the flexibility of defining different interpretable goals (6 different types explored in the paper) highlights the versatility of the approach;\n- the additional benefits in terms of plan reliability estimation (Appendix E) are significant;\n- the paper showcases how powerful and useful a good \"imitative model\" can be, therefore, reinforcing the interest of the research community in the important topic of off-line learning from large datasets of demonstrations (without requiring costly on-line data collection).\n\nKey reason 2: thorough experimental evaluation with convincing results.\n- the experimental protocol used is the standard one on CARLA and the results are state of the art;\n- the comparison with related works, including recent ones, is thorough and well explained;\n- the additional claims regarding robustness are substantiated by multiple experiments.\n\nSuggested improvements:\n- Not needing reward engineering is a major claim of this approach, but it seems that constructing goal likelihoods could be seen as a form of reward engineering, no? Table 3 indeed reports significant performance differences (absolute and relative) depending on how the goals are specified, especially in dynamic environments. As the experiments aim at maximizing the same performance metrics, is there a preferred goal type that works well across all experiments? If not, how is \"goal definition\" different than \"reward engineering\"?\n- Could the authors please include a variance analysis (using different seeds) in Tables 3 and 4? Previous papers have reported high variance in similar settings (cf., Codevilla et al 2019), and this is a common issue in IL/RL.\n- How important is knowing \\lambda (traffic light state) perfectly in practice? Can the robustness to noise in \\lambda be experimentally assessed? I would also clarify in section 4 and Table 2 that other methods do not use \\lambda (the traffic light state), which is a signal very strongly correlated with the \"ran red light\" metric.\n- More generally, what is the robustness of this approach to uncertainty / noise in \\phi? Although it is typically available (as the authors mentioned) it is never perfect in practice. Can this be handled in a principled probabilistic way as an extension of the current formulation?\n- The current model does not factor the influence of the agent on its environment (\\phi := \\phi_{t=0}). Is this framework limited to open loop planning, or does this open interesting future research directions towards closing the loop? It seems to be a key open problem to at least discuss in Section 5.\n\nAdditional Feedback:\n- Figure 5 is confusing, not sure it adds much value to the paper;\n- typos in Appendix (\"pesudocode\", \"baselines that predicts\", \"search search\").\n\n ## Update post rebuttal\n\nThanks to the authors' excellent replies and my initial inclination towards strong accept, I am happy to bump my score to 8. The authors did an excellent job, their rebuttal is on point, not avoiding hard questions, running additional requested experiments (incl. in a clever way for the most computationally expensive ones), and showing clear insights in future steps. Great job!", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xBD1MnKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose imitative models that learns goal-based probabilistic models of expert demonstrations, and use this to perform test-time planning and control of certain goal-directed behavior. The paper demonstrates extensive and impressive experiments on the CARLA simulator and outperforms existing approaches in the success metric.\n\nThe idea is quite simple: given a set of states, learn a probabilistic model that assigns high probability likelihood to expert behavior. After training, inference is performed to optimize the likelihood of a goal according to an expert prior. The paper discusses extensively how goal-based likelihood functions would be designed for autonomous driving, and the architectures for good q(s|\\phi) in detail, which are of engineering importance in self-driving applications. While I believe the method described might not be significantly novel technically, I believe the paper made nice contributions in terms of the autonomous driving application.\n\nMinor comments:\n\t- It seems like the term \"state\" is used to represent the agent's location on the ground plane, which is also not technically the entire state information?\n\t- Is the argmax in (1) a strict equality? I guess you would assume q(s|\\phi) = p(s|\\phi) for this to be always true?\n\t- How many iterations does it take to converge in Algorithm 2? It seems you would need to updates z{1:T} for all newly encountered goals? \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1x_4iWTYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper takes a model-based approach to \"imitation learning\". It first learns a (flow) \"model\" to assign likelihoods to trajectories being expert-like as well as sample expert-like trajectories. This is then combined with an estimate for trajectories being certain goal conditioned, where the goals come from a route planner. Results are shown in the context of autonomous driving in the CARLA simulator with a PID controller tracking the open-loop plan from the planner.\n\nOne major issue with the paper is that all the main contribution seem to be in the appendix. If I were to summarize it, the paper introduces an interesting way of integrating two different experts to perform learning from demonstration. On the one hand, you have the A* algorithm as an expert providing what waypoints to follow. On the other hand, you have the expert demonstrating how to drive around on the road. The question becomes, how do we reconcile these \"experts\" acting at different abstraction levels so that our system works in a more general setting. This kind of a setting is definitely not as general as the paper tries to make it out to be, but nevertheless it's reasonably broad and useful.\nThe idea to of different subsystems for route planning and path planning are not new, but the way it's done here does seem interesting to me.\n\nNote that Sec 3 related work CILS description seems hard to understand. Moreover, the claim \"MBRL can also plan, but with a one-step predictive model of possible dynamics\", seems incorrect. One can do multi-step predictions and use those to do model-based RL as well. So it's unfair to claim that this other approach is MBRL and yours is not on that basis. \n\nExperiments are interesting although quite dense. It's unclear what the initial state distribution look like for these experiments though. I'm guessing CARLA is otherwise a deterministic simulator?\n\nNow I am going to do something that you are not supposed to do as a reviewer. Tell the authors the paper they should have written, rather than the paper that was submitted. Apologies for that.\nThe terminology of Inverse RL, Imitation Learning and Apprenticeship Learning is a mess in the literature unfortunately and can't blame you not trying to fix it in this paper. However the community would benefit if don't overload these terms. Although all of these fall under learning from demonstrations, it's useful to restrict Inverse RL as trying to figure out the reward function being optimized by an expert, imitation learning as learning the exact behavior of an expert (and therefore not doing better than the expert which are by definition optimal or being applicable to doing something other than the expert) while apprenticeship learning as learning from demonstrations to perform even better at the task even generalizing to other things than the demonstrations. With these definitions under the belt the work in this paper is better positioned as about apprenticeship learning. Learning a model from the expert demonstration about how the world works and then using the same for accomplishing different tasks for which there are _no demonstrations_ doesn't sound like imitation?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to build an 'imitative model' to improve the performance for imitation learning. The main idea is to combine the model-based RL type of work to the imitation learning approach. The model is trained using a probabilistic method and can help the agent imitate goals that were previously not easy to achieve with previous works.

Reviewers 2 and 3 strongly agree that the paper should be accepted. R3 has increased their score after the rebuttal, and the authors' response helped in this case. Based on reviewers score, I recommend to accept this paper.
Decision: Accept (Poster)