Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
Authors: Oliver Scheel
Luca Bergamini
Maciej Wolczyk
Błażej Osiński
Peter Ondruska
Venue: CoRL2021 Poster (2021)
Keywords: Self-driving
Learning from Demonstrations
Simulation

Abstract: In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients.
We train our proposed method on 100 hours of expert demonstrations on urban roads and show that it learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios - all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.

Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
Oliver Scheel, Luca Bergamini, Maciej Wołczyk, Blažej Osiński, Peter Ondruska
Woven Planet, Level 5
(firstname (astname)@xoven wwaereeepppaaet...eee pplanet.gobbal
Abstract: In this work we the first present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This achieved by building a differentiable data- driven simulator on top of perception outputs and high-fidelity HD maps of the area. allows us to synthesize new driving experiences from existing demon- strations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method 100 hours of expert demonstrations learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previ- ously demonstrated state-of-ther of-the-art art for urban driving scenarios all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.
urban roads and show that it
Keywords: Self-driving, Learning from Demonstrations, Planning, Simulation
Closed-loop training
state
action
Collected driving logs Pereception
Simulator
Driving policy
SDV deployment
Figure 1: Overview of the proposed closed-loop training method for learning driving policies. We leverage large amounts of expert demonstrations and mid-to-mid representations to build a differen- tiable simulator supporting fast policy learning. With this simulator, we effectively unroll model policies, and thus train the model closed-loop using a policy gradient method.
Introduction
Self-driving has the potential to revolutionize transportation and is a major field of Al applications.
Even though already in 1990 there were prototypes capable of driving on highways [1], technology still not widespread, especially in the context of urban driving. In the past decade, the availability of large datasets and high-capacity neural networks has enabled significant progress in perception [2, 3] and the vehicles' ability to understand their surrounding environment. Self-driving decision making, however, has seen very little benefit from machine learning or large datasets. State-of- the-art planning systems used in industry [4] still heavily rely on trajectory optimisation techniques with expert-defined cost functions. These cost functions capture desirable properties of the future vehicle path. However, engineering these cost functions scales poorly with the complexity of driving situations and the long tail of rare events.
Due to this, learning a driving policy directly from expert demonstrations is appealing, since perfor- mance scales to domains by adding data rather than via additional human engineering effort. 5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

In this paper we focus specifically on learning rich driving policies for urban driving from large amounts of real-world collected data. Unlike highway driving [5], urban driving requires perform- ing a variety of maneuvers and interactions with, e.g., traffic lights, other cars and pedestrians.
Recently, rich mid-level representations powered by large-scale datasets [6, 7], HDD-maps and high- performance perception systems enabled capturing nuances of urban driving. This led to new meth- ods achieving high performance for motion prediction [8. 9]. Furthermore, [10] demonstrated that leveraging these representations and behavioral cloning with state perturbations leads to learning robust driving policies. While promising, difficulty of this approach lies in engineering the pertur- bation noise mechanism required to avoid covariate shift between training and testing distribution.
Inspired by this approach, we present the first results on offline learning of imitating driving policies using mid-level representations, a closed-loop simulator and a policy gradient method. This formu- lation has several benefits: it can successfully learn high-complexity maneuvers without the need for perturbations, implicitly avoid the problem of covariate shift, and directly optimize imitation as well as auxiliary costs. The proposed simulator is constructed directly from the collected logs of real- world demonstrations and HD maps of the area, and can synthesize new realistic driving episodes from past experiences (see Figure for an overview of our method). Furthermore, for training on large datasets reducing the computational complexity is paramount. We leverage vectorized repre- sentations and show how this allows for computing policy gradients quickly using backpropagation through time. We demonstrate how these choices lead to superior performance of our method over the existing state-of-the of-the-art art in imitation learning for real-worle self-driving planning in urban areas.
Our contributions are four-fold:
The first demonstration of policy gradient learning of imitative driving policies for complex urban driving from a large corpus of real-world demonstrations. We leverage closed- loop simulator and rich, mid-level vectorized representations to learn policies capable of performing a variety of maneuvers.
A new differentiable simulator that enables efficient closed-loop simulation of realistic driving experiences based on past demonstrations, and quickly compute policy gradients by backpropagation through time, allowing fast learning.
A comprehensive qualitative and quantitative evaluation of the method and its performance compared to existing state-of-the-art, We show that our approach, trained purely in simula- tion can control real-world self-driving vehicle, outperforms other methods, generalizes well and can effectively optimize both imitation and auxiliary costs.
Source code and data are made available to the public'.
2
Related Work
In this section we summarize different approaches for solving self-driving vehicle (SDV) decision- making in both academia and industry. In particular, we focus on both optimisation- and ML-based systems. Furthermore, we discuss the role of representations and datasets as enablers in recent years, to tackle progressively more complex Autonomous Driving (AD) scenarios.
Trajectory-based optimization is still dominant approach used in industry for both highway and urban-driving scenarios. It relies on manually defined costs and reward functions that describe good driving behavior. Such cost can then be optimized using a set of classical optimization techniques (A* [11], RRTs [12), POMDP with solver [13], or dynamic programming [14]). Appealing prop- erties of these approaches are their interpretability and functional guarantees, which are important for AD safety. These methods, however, are very difficult to scale. They rely on human engineer- ing rather than on data to specify functionality. This becomes especially apparent when tackling complex urban driving scenarios, which we address in this work.
Reinforcement learning (RL) [15] removes some complexity of human engineering by providing a reward (cost) signal and uses ML to learn an optimal policy to maximize it. Directly providing the reward through real-time disengagements [16], however, is impractical due to low sample- efficiency of RL and the involved risks. Therefore, most approaches [17] rely on constructing a
'Code and video available https::/plamiingl55kttt.o

---

simulator and explicitly encoding and optimising a reward signal [18]. A limiting factor of these approaches is that the simulator often is hand-engineered [19, 20], limiting its ability to capture long-tail real-world scenarios. Recent examples of sim-to-real policy transfer (e.g. [21], [22], [23]) were not focused on evaluating scenarios typical to urban driving. in particular interacting with other agents. In our work, we construct the simulator directly from real-world logs through mid- level representations. This allows training in a variety of real-world scenarios with other agents present, while employing efficient policy-gradient learning.
Imitation learning (IL) and Inverse Reinforcement Learning (IRL) [24, 25] are more scalable ML approaches that leverage expert demonstrations. Instead of learning from negative events, aim
directly copy expert behavior or recover the underlying expert costs. Simple behavioral cloning was applied already back in 1989 [26] on rural roads, more recently by [27] on highways and [28] in urban driving. Naive behavioral cloning, however, suffers from covariate shift [24). This issue has been successfully tackled for highway lane-following scenarios by reframing the problem as classification task [29] or employing simple simulator [5], constructed from highway cameras. We take inspiration from these approaches but focus on the significantly more complex task of urban driving. Theoretically, our work is motivated by [30], as we employ a similar principle of generating synthetic corrections to simulate querying expert. Due to this, identical proven guarantees hold for our method, namely the ideal linear regret bound, mitigating the problem of covariate shift. Adversarial Imitation Learning comprises another important field [31, 32, 33]. but, to the best of our knowledge, has seen little application to autonomous driving and no actual SDV deployment yet.
Neural Motion Planners are another approach used for autonomous driving. In [34] raw sensory input and HD-maps are used to estimate cost volumes of the goodness of possible future SDV positions. Based on these cost volumes, trajectories can be sampled and the lowest-cost one is selected to be executed. This was further improved in [35], where the dependency on HD-maps was dropped. To the best of our knowledge, these promising methods have not yet been demonstrated to drive a car in the real-world though.
niid-representationn and the availability of large-scale real-world AD datasets [6. 7] have been major enablers in recent years for tackling complex urban scenarios. Instead of learning policies directly from sensor data, the input of the model comprises the output of the perception system as well as an HD map of the area. This representation compactly captures the nuance of urban scenarios and allows large-scale training on hundreds or thousands of hours of real driving situations. This led new state-of-the-an solutions for motion forecasting [8, 9]. Moreover, [10] demonstrated that using mid-represeenttions, large-scale datasets and simple behavioral cloning with perturbations [36] can scale and learn robust planning policies. The difficulty of this approach, however, is in engineering the noise model to overcome the covariate shift problem. In our work we are inspired by this approach, but attempt to learn robust policies using policy gradient optimisation [37] featuring unrolling and evaluating the policy during training. This implicitly avoids the problem of covariate shift and leads to superior results. This approach is, however, more computationally expensive and requires simulator. To solve this, we show how a fast and powerful simulator can be constructed directly from real-world logs enabling scalability of this approach.
Data-driven simulation. A realistic simulator is useful for both training and validation of ML models. However, many current simulators (e.g. [19, 38]) depend on heuristics for vehicle control and do not capture the diversity of real world behaviours. Data-driven simulators are designed to alleviate this problem. [23] created a photo-realistic simulator for training an end-to-end RL policy. [5] simulated a bird's-eye view of dense traffic on a highway. Finally, two recent works [39, 40] developed data-driven simulators and showed their usefulness for training and validating ML planners. In this work we show that a simpler, differentiable simulator based on replaying logs is effective for training.
3 Differentiable Traffic Simulator from Real-world Driving Data
In this section we describe a differentiable simulator S 'hat approximates new driving experiences based on an experience collected in the real world. This simulator is used during policy learning for the closed-loop evaluation of the current policy's 's performance and computing the policy gradient. As shown Section 5, differentiability is an important building block for achieving good results, especially when employing auxiliary costs.
3

---

Real world
S
S3
Simulator
Simulator
Figure 2: Differentiable simulator from observed real-world logs: based ground truth log (top row), we unroll a new trajectory corresponding to different SDV actions (e.g. given by a planner) in the simulator approximating the vectorized world representation (bottom row)
We represent the real-world experience 7 sequence of state observations S, around the vehicle over time:
== {51, ................................ Sr)
(1)
We use a vectorized representation based on [8], in which each state observation š, consists of collection of static and dynamic elements E .e...., around the vehicle pose Pt E SE2, with SE denoting the special Euclidean group. Static elements include traffic lanes, stop signs and pedestrian crossings. These elements are extracted from the underlying HD semantic map of the area using the localisation system. The dynamic elements include traffic lights status and traffic participants (other cars, buses, pedestrians and cyclists). These detected in real-time using the on-board perception system. Each element e/ includes pose of E SE relative to the SDV pose Pt. as well as additional features, such as the element type, time of observation, and other optional attributes, e.g. the color of associated traffic lights, recent history of moving vehicles, etc. The full details of this representation are provided in Appendix C.
Goal of the simulation is to iteratively generate a sequence of state observations {81, . 8T) that corresponds to a different sequence of driver actions 01, 02, aN in the sce- nario. This is done by first computing the corresponding SDV trajectory Pt, P2..P.....PP .P and then locally transforming states 1 12,52,..... ,
Updated poses of the SDV are determined by a kinematic model Pt+1 = [pr: ar), which is assumed to be differentiable. The state observation is then obtained by computing the new position of for each state element eş using transformation along the differences of the original and updated pose:
df के (p: p.).
(2)
See Figure 2 for illustrative example. It is worth noting that this approximation is effective if the distance between the original and generated SDV pose is not too large.
We denote performing these steps in sequence with the step-by-step simulation transition function $1+1 S(st. oz). Moreover, since both Equation (2) and vehicle dynamics f are fully differentiable, compute gradients with respect both the state (S.) and action (Sa). This is critical for the efficient computation of policy gradients using backpropagation through time as described in the next section.
Imitation Learning Using a Differentiable Simulator
In this part, we detail how we use the simulator S described in the previous section to learn a deterministic policy to drive using closed-loop policy learning.
We frame the imitation learning problem as minimisation of the LI pose distance L(st, a:)
lp Pelli between the expert and learner a sequence of collected real-world demonstrations 71, T2, ...: TN ng. Note that with slight abuse of notation we use the poses Pt, Pt here to refer to 3D vectors (x. y, 0), instead of roto-translation matrices in SF2 yielding the common I. norm

---

Policy trajectory
Expert trajectory
Policy sampling
Palicy gradient update
Figure 3: One iteration of policy gradient update. Given a real-world expert trajectory F we sample policy state 8, by unrolling the policy n for T steps. We then compute optimal policy update by backpropagation through time.
and loss. This can be expressed as discounted cumulative expected loss [41] on the set of collected expert scenarios:
J(T) = EE BYna Etnnaa))
(3)
Optimising this objective pushes the trajectory taken by the learned policy close as possible the one of the expert, as well as limiting the trajectory to the region where the approximation given by the simulator effective. In Appendix B further extend this include auxiliary cost functions with the aim of optimising additional objectives.
We can use any policy optimisation method [42, 43] to optimize Equation (3). However, given that the transition S(2,, 0z) differentiable, we can exploit it for more effective training that does not require separate estimation of value function. As shown in [31, 37, 44], this results into order of magnitude more efficient training. The optimisation process consists of repeatedly sampling pairs of expert and policy trajectories F, and computing the policy gradient Ja for these samples to minimize Equation (3). We describe both steps in detail the following subsections.
4.1 Sampling from a Policy Distribution
Algorithm 1: Imitation learning from expert demonstrations
In this subsection detail sampling pairs of expert (F) and corresponding policy trajectory (T) drawn from policy π.
Input: Expert policy samples
Fi, F2, :.: TN ~ T
Sampling expert trajectories consists of sim- ply sampling from the collected dataset of ex- pert demonstrations. To generate the policy sample we acquire an expert state 31 E f, and then unroll the current policy n for steps using the simulator S.
Output: Learned policy
π random:
for 7 Fg do
for p = to T do
d, T(s);;
8141 = S( at);
This naive method, however, introduces bias, as the initial state of the trajectory is always drawn from the expert Te and not from the policy distribution π. As shown in Appendix B, this results in the under-performant method. To remove this bias we discard the first K timesteps from both trajectories and use only the remaining T K timesteps to estimate the policy gradient Jo as described next (see Figure 3 for a visualization).
end
JT+1 = 0;
J T++ 0;
for
T downto K do
of the
J Log + Thg R + JJ ( S.m,);
J = LaTo + 7 ([t+1 1. + J(+1):
end
π tradient_updater (1, JKK:
end
5

---

4.2 Computing Policy Gradient J9
Here we describe the computation of the policy gradient J9 around the rollout trajectory т $1, 01 $2, ................................ SI. ar given by the current policy. This gradient can be computed for determin- istic policies using backpropagation through time leveraging the differentiability of the simulator
S. Note that we denote partial differentiation with subscripts, i.c. Is - 0g(z,...) We follow the formulation in [37] and express the gradient by pair of recursive formulas:
J = Ls + Ta + 7 (S. + SaTs),
(4)
J = LaTO + ( - -g+1)
(5)
The resulting algorithm is outlined in Algorithm 1 and illustrated in Figure 3. It can be implemented simply as one forward pass of length and one backward pass of length - K. To compute the policy gradient equations (4) and (5) recursively from T to f K and use it to update policy parameters 8.
Experiments
In this section we evaluate our proposed method and benchmark it against existing state-of-the-an systems. In particular, we are interested in: its ability learn robust policies dealing with various situations observed in the real world; its ability to tailor performance using auxiliary costs; the sensitivity of key hyper-parameters; and the impact on performance with increasing amounts of training data. Additional results can be found in the appendix and the accompanied video.
5.1 Dataset
For training and testing our models the Lyft Motion Prediction Dataset [6]. This dataset was recorded by a flect Autonomous Vchicles and contains samples of real-world driving on a complex, urban route in Palo Alto, California. The dataset captures various real-world situations, such as driving in multi-lane traffic, taking turns, interactions with vehicles at intersections, etc. Data was preprocessed by a perception system, yielding the precise position of nearby vehicles, cyclists and pedestrians over time. addition, a high-definition map provides locations of lane markings, crosswalks and traffic lights. All models trained on 100h subset, and tested on 25h. The training dataset is identical to the publicly available one, whereas for the sake of execution speed for testing we random, but fixed, subset of the listed test dataset, which roughly I in size.
5.2 Baselines
We compare our proposed algorithm against three state-of-the-ee baselines:
Naive Behavioral Cloning (BC): we implement standard behavioral cloning using our vec- torized backbone architecture. We do not use the SDV's history as an input to the model to avoid causal confusion (compare [10]).
Behavioral Cloning Perturbations (BC-perturb): ChauffeurNet [10] using our backbone network. As in the original paper, we add noise in the form of perturbations during training, but do not employ any auxiliary losses. We test two versions: without the SDV's history, and using the SDV's history equipped with history dropout.
re- re-implement vectorized version
Multi-step Prediction (MS Prediction): we apply the meta-learning framework proposed in [30] train our vectorized network. We observe that version of this algorithm can conveniently be expressed within our framework: we obtain it by explicitly detaching gra- dients between steps (i.e. ignoring the full differentiability of our simulation environment). Differently from the original work [30], we do not save past unrolls as new dataset samples over time.
5.3 Implementation
Inspired by [8, 45), we use a graph neural network for parametrizing our policy. combines PointNet-like architecture for local inputs processing followed by attention mechanism for global

---

Configuration
Collisions
Imitation
Model
SDV history
Front
Side
Rear
Off-road
L2
Comfort
11K
BC
79 + 23
395 = 170
997 + 74
1618 + 459
1.57 + 0.27
93K +: 3K
3 = 601
BC periiub
2
56 1: 6
411 146
82 11
0.74 1 0.01
203K 1 6K
567 128
BC periurb
14 +:
73 7
678 1: 11
77 1:
0.77 1 0.01
636K 1: 22K
843 1: 6
MS Prediction
55 4
141
0.46 + 0.02
595K 49K
341 +
Ours
13
46 5
101 = 13
97
0.42 # 0.00
637K # 41K
260 = 9
Table I: Normalized metrics for all baselines and our method reporting mean and standard de-
viation for each obtained from 3 runs. For all, lower is better. Our method overall yields best performance and lowest IIK.
reasoning. contrast to [8], we use points instead of vectors. Given the set of points correspond- ing to each input element, employ 3 PointNet layers to calculate 128- 128-dimensional feature descriptor. Subsequently, a single layer of scaled dot-product attention performs global feature ag- gregation, yielding the predicted trajectory. We found K 20 and T = 32 to work well, i.e. use 20 timesteps for the initial sampling and effectively predict 12 trajectory steps. y is set to 0.8. In total, our model contains around 3.5 million trainable parameters, and training takes 30h 32 Tesla V100 GPUs. For more details we refer to Appendix C.
For the vehicle kinematics model f we This allows for a fair comparisons with the baselines as both BC-perturb and MS Prediction assume the possibility of arbitrary pose corrections. Other kinematics models, such unicycle or bicycle models, could be used with our method as well.
unconstrained model Pt+1 Pt + ag with de E SE2-
All baseline methods share the same network backbone as ours, with model specific differences as described above - and BC and BC-perturb predicting a full T-step trajectory with a single forward, while MS Prediction and ours are calling the model T times. To ensure a fair comparison, also for MS Prediction we use our proposed sampling procedure, i.e. use the first K steps for sampling only. We train all models for 61 epochs with a learning rate of 10-4, and drop it to 10-5 after 54 epochs. We note that we achieve best results for our proposed method by disabling dropout, and hypothesize this is related to similar issues observed for RNNs [46].
We refer the reader to Appendix B for ablations on the influence on data and sampling.
5.4 Metrics
We implement the metrics describe below to evaluate the plamning performance. These capture key imitation performance, safety and comfort. In particular, we report the following. which are normalized if applicable per 1000 miles driven by the respective planner:
L2: L.2 distance to the underlying expert position the driving log meters.
Off-road events: we report a failure if the planner deviates more than 2m laterally from the reference trajectory this captures events such as running off-road and into opposing traffic.
Collisions: collisions of the SDV with any other agent, broken down into front, side and rear collisions w.r.t. the SDV.
we monitor the absolute value of acceleration, and raise failure should this exceed m/s?
Comfort:
IIK: accumulate safety-critical failures (collisions and off-road events) into key metric for ease of comparison, namely Interventions per 1000 Miles (IIK).
5.5 Imitation Results
We evaluate our method and all the baselines by unrolling the policy on 3600 sequences of 25 seconds length from the test set and measure the above metrics.
Table reports performance when all methods are trained to optimize the imitation loss alone. Behavioral cloning yields a high number of trajectory errors and collisions. This is expected, this approach known to suffer from the issue of covariate shift [24]. Including perturbation during training dramatically improves performance as it forces the method to learn how to recover from
7

---

drifting. We further observe that MS Prediction yields comparable results for many categories, while yielding less rear collisions. We attribute this to the further reduction of covariate shift when compared to the previous methods: the training distribution is generated on-policy instead of being synthesized by adding noise. Finally, our method yields best results overall. It is worth noting that all models share a high number of comfort failures, due to the fact that they are all trained for imitation performance alone, which does not optimize for comfort, but only positional accuracy of the driven vehicle which we address in the appendix.
5.6 In-car Testing
In addition above stated simulation results, we further deployed our planner on SDVs in the real. For this, Ford Fusion equipped with 7 camera, 3 LiDAR and 11 Radar sensors was employed. The sensor setup thus equals the one used for data collection, and during road-testing our perception and data-processse
stack is run in real-time to generate the desired scene representation on the fly. For this, vehicles are equipped with Nvidia 2080 Tls. Experiments were conducted on private test track, including other traffic participants and reproducing challenging driving scenarios. Furthermore, this track was never shown to the network before, and thus offers valuable insights into generalization performance. Figure 5 shows model successfully crossing signaled intersection, for more results we refer to the appendix and our supplementary video.
Conclusion
In this work have introduced method for learning an autonomous driving policy in urban setting, using closed-loop training, mid-level representations with a data-driven simulator and a large corpus of real world demonstrations. We show this yields good generalization and performance for complex, urban driving. In particular, it can control real-world self-driving vehicle, yielding better driving performance than other state-of-the-an ML methods.
We believe this approach can be further extended towards production grade real-world driving re- quirements of L4 and L5 systems in particular, for improving performance in novel rarely scenarios and to increase sample efficiency, allowing further scaling to millions of hours of driving.
Figure 4: Qualitative results of our method controlling the SDV. Every row depicts two scenes, images are 2s apart. The SDV is drawn in red, other agents in blue and crosswalks in yellow. Traffic lights colors are projected onto the affected lanes. Best view screen.
Figure 5: Front-camera footage of our planner stopping for red light, and subsequently crossing the intersection when the signal turns green (images from left to right, recorded several seconds apart).

---

Acknowledgments
We would like to thank everyone at Level 5 working on data-driven planning, in particular Sergey Zagoruyko, Yawei Ye, Moritz Niendorf, Jasper Friedrichs, Li Huang, Qiangui Huang, Jared Wood, Yilun Chen, Ana Ferreira, Matt Vitelli, Christian Perone, Hugo Grimmett, Parth Kothari, Stefano Pini, Valentin Irimia and Ashesh Jain. Further we would like to thank Alex Ozark, Bernard Barcela, Alex Oh, Ervin Vugdalic, Kimlyn Huynh and Faraz Abdul Shaikh for deploying our planner to SDVs in the real.
References
[1] E. Dickmanns. Dynamic vision for perception and control of motion. 2010.
[2] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.
[3] R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
[4] H. Fan, F. Zhu, C. Liu, L. Zhang, I. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, Baidu apollo em motion planner. ArXiv, 2018.
[5] M. Henaff, A. Canziani, and Y. LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. ArXiv, 2019.
[6] J. Houston, G. Zuidhof, L. Bergamini, Y. Ye, A. Jain, S. Omari, V. Iglovikov, and P. Ondruska. One thousand and one hours: Self-driving motion prediction dataset. Conference on Robot Learning (CoRL), 2020.
[7] M.-F Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, P. C. De Wang, S. Lucey, D. Ramanan, and Hays. Argoverse: 3d tracking and forecasting with rich maps supplemen- tary material. Int. Conf. Computer Vision and Pattern Recognition (CVPR).
[8] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectomet: Encoding hd maps and agent dynamics from vectorized representation. In Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[9] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. 2020.
[10] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. 12 2018.
[11] J. Ziegler, M. Werling, and J. Schroder. Navigating car-like robots in unstructured environ- ments using an obstacle sensitive cost function. In Intelligent Vehicles Symposium, 2008.
[12] J. K. Jr. and S. M. Lavalle. Rrt-connect: An efficient approach to single-query path planning. In Int. Conf. on Robotics and Automation, 2000.
[13] T. Bandyopadhyay, K. S. Won, E. Frazzoli, D. Hsu, W. S. Lee, and D. Rus. Intention-aware motion planning. In E. Frazzoli, T. Lozano-Perez, N. Roy, and D. Rus, editors, Algorithmic Foundations of Robotics X, 2013.
[14] M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, Dolgov, S. Ettinger, D. Haehnel, T. Hilden, G. Hoffmann, B. Huhnke, D. Johnston, S. Klumpp, D. Langer, A. Levandowski, Levinson, J. Marcil, D. Orenstein, J. Paefgen, Penny, A. Petrovskaya, M. Pflueger, G. Stanek, D. Stavens, A. Vogt, and S. Thrun. Junior; The Stanford Entry in the Urban Challenge. 2009.
[15] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[16] A. Kendall, Hawke, D. Janz, Mazur, D. Reda, J.MM. Allen, V.DD Lam, A. Bewley, and
A. Shah. Learning to drive in day. In Int. Conf. on Robotics and Automation (ICRA), 2019.

---

[17] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey. Transactions on Intelligent Transportation Systems, 2021.
[18] S. Shalev-Shwartz S. Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. ArXiv, 2016.
[19] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and Koltun. CARLA: An open urban driving simulator. In /st Annual Conference on Robot Learning, 2017.
[20] P. A. Lopez, M. Behrisch, L. Bieker-Wall J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, Rummel, Wagner, and E. WieBner. Microscopic traffic simulation using sumo. Int. Conf. on Intelligent T Transportation Systems (ITSC), 2018.
[21] Y. You, X. Pan, Z. Wang, and C. Lu. Virtual to real reinforcement learning for autonomous driving. 2017.
[22] B. Osiński, A. Jakubowski, P. Zięcina, P. Mitos, C. Galias, S. Homoceanu, and H. Michalewski. Simulation- baased reinforcement learning for real-world autonomous driving. In Int. Conf. on Robotics and Automation (ICRA), 2020.
[23] A. Amini, I. Gilitschenski, Phillips, Moseyko, R. Banerjee, S. Karaman, and D. Rus. Learning robust control policies for end-to-end autonomous driving from data-driven simula- tion. Robotics and Automation Letters, 2020.
[24] Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Fourteenth Int. Conf. on Artificial Intelligence and Statistics, 2011.
[25] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In National Conference Artificial Intelligence, 2008
[26] D. A. Pomerleau. ALVINN: An Autonomous Land Vehicle in Neural Network. 1989.
[27] M. Bojarski, D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, Zhao, and K. Zieba. End to end learning for self-driving cars. ArXiv, 2016.
[28] J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, and A. Kendall. Urban driving with conditional imitation learning. In Ini. Conf. on Robotics and Automation (ICRA), 2020.
[29] M. Bojarski, C. Chen, J. Daw, A. Degirmenci, J. Deri, B. Fireer, B. Flepp, S. Gogri. J. Hong,
L. Jackel, et al. The nvidia pilotnet experiments. arXiv preprini urXiv:2010.08776 2020.
[30] A. Venkatraman, M. Hebert, and J. Bagnell. Improving multi-step prediction of learned time series models. In AAAI, 2015.
[31] N. Baram, O. Anschel, I. Caspi, and Mannor. End-to-end differentiable adversarial imitation learning. In Int. Conf. on Machine Learning, 2017.
[32] Ho and S. Ermon. DeXiv:1606.0344 2016.
Generative adversarial imitation learning. arXiv preprint
[33] R. P. Bhattacharyya, B. Wulfe, D. J. Phillips, A. Kuefler, J. Morton, R. Senanayake, and M. J. Kochenderfer. Modeling human driving behavior through generative adversarial imi- tation learning. ArXiv, 2020.
[34] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. Ins. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[35] S. Casas, A. Sadat, and R. Urtasun. Mp3: A unified model to map, perceive, predict and plan. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14403-14412, 2021.
10

---

[36] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In 1st Annual Conference on Robot Learning, 2017.
[37] N. Heess, G. Wayne, D. Silver, T. Lillicrap. T. Erez, and Y. Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, 2015.
[38] E. Leurent. An environment for autonomous driving decision-making 2018.
[39] L. Bergamini, Y. Ye, O. Scheel, I.. Chen, C. Hu, L.. Del Pero, B. Osinski, H. Grimmett, and
Ondruska. Simnet: Learning reactive self-driving simulations from real-world observations. Int. Conf. on Robotics and Automation, 2021.
[40] S. Suo, S. Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. 2021.
[41] A. Y. Ng. Russell, et al. Algorithms for inverse reinforcement learning. In Icml, 2000.
[42] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1oXiv:1707006333 2017.
[43] V. Konda and J. Tsitsiklis. Actor-critic algorithms. In SIAM Journal on Control and Optimiza- tion, 2000.
[44] W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell, Deeply Aggre VaTeD: Differentiable imitation learning for sequential prediction. Int. Conf. on Machine Learning.
[45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, 2017.
[46] W. Zaremba, Sutskever, and O. Vinyals. Recurrent neural network regularization. 2014.


Reviews:
[{"review_id": "Ykhgkk_2Egn", "reviewer": "Reviewer_KT1F", "summary": "The paper proposes a simulation-based, imitation learning-based approach for learning policies for urban driving scenarios from real-world human driving trajectories. To train the policies, a simulator is developed to replay traffic and simulate the ego vehicle, where the ego vehicle action is governed by a policy model utilizing PointNet and Multi-Head Attention layers. The policy model takes HD map, perceived objects' polygons, and the ego vehicle polygon as inputs and outputs a series of position coordinates and orientations as a trajectory. The model is trained by minimizing the error between the generated trajectories and those of experts. The results are compared with behavioral cloning and multi-step prediction methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is on presenting a differentiable traffic simulator from real-world driving data, which can potentially be used to generate synthesized driving scenarios and as a testbed for autonomous driving applications.  However, the paper does not provide sufficient details on the simulator and how the traffic is managed in the simulator. It seems that the traffic is simulated simply by replaying data. This could be problematic in many autonomous driving scenarios including scenarios, in which the ego vehicle behavior is slightly different from the expert demonstration.  For example, consider a driving case where the speed of the ego vehicle is marginally lower than the expert driver when stopping at a red traffic light. Without adjusting the traffic, this could result in a rear-end collision. This can become more critical when there is a significant difference between the policy and demonstration. For example, consider a case where the policy decides to make a valid lane change instead of remaining within the lane to follow the demonstration.  Given these cases, the application of the simulator seems to be restricted without considering a proper traffic management system. Additionally, the simulator's differentiablity seems to be narrowed down to the kinematic model of the ego vehicle, which is further reduced to a simple motion model. The simulator is not detailed adequately to judge whether perception inaccuracy or collision can lead to discontinuity or not. For example, does the simulator remains differentiable in case of collision or disappearance of an object? Lastly, it's not clear if the result is generalizable to different maps or scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The simulator is not detailed enough to judge its usability and impact on autonomous driving research and if there are any differences between the presented simulator and other simulators (e.g. [1]). The imitative policy seems to be just another model without being properly validated through ablation tests, generalizability results, and sufficient comparison results with SOTA. While less important, the provided comparison results also do not show a significant/clear improvement to BC. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "1. The comfort method is based on longitudinal acceleration. It's recommended to consider jerk and/or lateral acceleration instead as they are more representative of comfort in driving.\n2. An off-road event threshold seems to be too large (4 m). Considering that lane width ranges between 3-4 m and vehicles usually drive in the center of lanes, the threshold could be reduced to 1.5-2 m to capture off-lane events. \n3. Referring to Table 1, BC and MS Prediction results seem to be very close to that of the proposed method when considering front and side collisions, but significantly higher for rear collisions. The reason behind this needs to be elaborated on in the paper. Should rear collisions be considered a failure?\n4. The comparison results can be extended to more recent works including IRL/IM papers mentioned in the related work section.\n5. There is no result to judge if the learned policy can generalize to different maps or scenarios. Further tests need to be provided to check the performance of the policy in a novel, but a similar map.\n6. The simulator differentiably needs to be discussed in the case of collision, sudden appearance/disappearance of objects.\n7. The related works need more recent methods to cover the whole problem including other end-to-end methods.\n8. The difference between the presented simulator and the differentiable simulator presented in [1] needs to be highlighted.\n9. Line 165-167, \"To remove this bias we discard the first K timesteps ...\", an experiment showing a comparison between with or without discarding K timesteps is helpful.\n10. The introduction discusses covariant shifts between training and test distributions, but it is not clear how the proposed approach has addressed this problem."}}, {"review_id": "IEDh-lwdkeT", "reviewer": "Reviewer_wo2G", "summary": "The present work proposes a novel differentiable data-driven scenario simulation technique that generates driving scenarios based on real-world data. This simulator is incorporated as a differentiable module in a policy gradient-based driving policy learning task. The authors evaluate the resulting policy learning system in comparison to several policy learning baselines on the Lyft Motion Prediction Dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__What I learned__\n- I learned how differentiable driving scenario generators can be integrated into a policy learning pipeline.\n\n__Strengths__\n- Data-driven simulation engine improves the data efficiency of the training process.\n- Differentiability of the simulator allows its use as a neural network module in a more complex fully differentiable pipeline.\n\n__Weaknesses__\n- The positioning and motivation of the work is not fully clear, are you focussing more on the simulator or the policy learning aspect? On the policy learning side, what is the ultimate goal of the paper? Mainly basic research in embodied AI for mobility? Or actual deployment of learned driving policies for Self-Driving Vehicles (SDVs)?\n- Depending on the answers to the former, different (emphasis of the) evaluations may be necessary. For example, focusing on improved navigation behavior of real-world SDVs might require comparisons with methods based on trajectory-based optimization or other AV planning techniques such as Multi-Policy Decision Making (Cunningham et al., 2015). Another evaluation-related issue is that it is done purely on a dataset rather than on any actual driving platform or at least in some video-game environment.\n- Finally, the simulator itself is not compared with other scenario simulators such as TrafficSim.\n- Some implementation details are missing, for example how was the data split (did you simply use the default split given in the dataset?).\n-  The code is unfortunately not available at the time of submission. I still think it is great that the authors plan on releasing it.\n- At times some smaller aspects are hard to understand (some examples are given as questions in the issues section). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I see this work as going in a very interesting direction. My current objections are mainly motivation and evaluation related: The authors should also better clarify the motivation and positioning of the work. And the work needs to also evaluate the simulator in comparison to state-of-the-art. While the latter point may require significant work, I believe the motivation-related issues to be easily resolvable.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "__Main Paper__\n- l. 79 / 80 What do you mean by \"sim-to-real transfer\" here? E.g. in the case of [23], the policy is not even retrained for the actual transfer.\n- l. 129 & eq (2): In the current form of eq. (2), when the original and updated pose are the same, it seems like the new position becomes zero. Unless I am misunderstanding something, this does not look right.\n- l. 138 do I understand correctly, that the simulator mainly allows for local perturbations of the expert trajectories rather then completely new scenarios in complex scenes such as intersections?\n- l. 163: It is not clear to me why bias is introduced if the initial state of the trajectory is sampled from the expert? What characteristics of the expert driver's initial position are biased? Maybe something like speed?\n- l. 198.: Could you provide implementation details on how exactly ChauffeurNet was modified (i.e. more details on your vectorization process for ChauffeurNet).\n\n__Appendix__\n- l. 16 How is the type embedding added? Simply via a one-hot representation?"}}, {"review_id": "F4OvhIvdAC", "reviewer": "Reviewer_oEVn", "summary": "This paper constructs a simulation environment based on real traffic data and train a traffic agent by means of imitation learning. To alleviate the covariate shift phenomenon, this paper introduces the “Differentiable simulator” method. The authors compare the performances of their method with some classical methods of imitation learning and show that their method has better performance in some indicators such as collision rate. In addition, the paper also carries out comparative tests on some reinforcement learning kinds of auxiliary rewards.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths： \nThe first is the investigated problem is very meaningful for autonomous driving, which is how to imitate a human-like traffic agent policy in a complex traffic scenario from the real dataset. \nThe second is that this paper combines some current mainstream methods such as pointNet and differentiable forward loss for policy to improve the performance of the imitation learning.\n\nThe biggest weakness of the approach is, in my opinion, the lack of novelty. Indeed, all the core ideas of the contribution have been already proposed in existing methods. The idea of using differentiable simulator for reducing covariate shift was already proposed, with exactly the same loss style but different in the concrete form of calculation (this paper even use a more simplified form of Ls and La term, which in [35] are represent the backpropagation item towards policy parameter.) in [35] (in the paper). The use of reinforcement learning kinds of reward for helping revise the imitation learning policy behavior was already used in [1]. In addition, the baseline methods compared in this paper, One is behavior cloning inherently flawed in terms of covariate shift problem, which makes this comparison seem insufficiently persuasive. Therefore, the work appears to be very incremental at the moment, with only very small variations of existing methods.\n[1] Bhattacharyya, R. , et al. \"Modeling Human Driving Behavior through Generative Adversarial Imitation Learning.\" (2020).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the novelty is not enough, the investigated problem is very meaningful for autonomous driving. And the experimental result is convincing. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I think that this paper needs more empirical evidence that the proposed method outperforms other SOTA algorithm in imitation learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "7R-2olYaiM", "reviewer": "Reviewer_q8Fh", "summary": "The paper introduces a differential simulator for self-driving in urban areas and uses it to build an imitation learning approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n* The paper introduces a differentiable simulator for self-driving.\n* The paper evaluates a method for imitation learning based on differentiation through the dynamics model of the simulator.\n* The method is evaluated against several baselines.\n* The authors are planning to share their open-source implementation for training and evaluation on the Lyft motion planning dataset.\n\n### Weaknesses \n* The paper lacks algorithmic novelty. Planning through a model has already been explored in prior work, for example, in Universal Planning Networks (Srinivas etal., 2018).\n* The set of the baseline is not adequate. I would encourage the authors to compare with Adversarial Imitation Learning methods as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend the paper for acceptance solely for the reason that the authors promise to share their code. Having a good codebase to build further research on can be just as important for advancing the topic as introducing novel algorithms. After a quick googling, I wasn't able to find another repository that performs imitation learning on the Lyft dataset.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "As think that the paper suffers from a limited comparison to the prior work. Also, the method for imitation learning introduced in the paper has limited novelty. However, I would strongly encourage the authors to continue their work since even benchmarking IL methods for self-driving and sharing their implementations is a valuable contribution on its own."}}]

Author rebuttal:

Meta-review: The authors provided thorough responses that addressed the reviewers' concerns. Following the authors' responses, all reviewers are generally positive. The AC endorses the reviewers' recommendation to accept the submission.
Decision: Accept (Poster)