CCIL: Context-conditioned imitation learning for urban driving
Authors: Ke Guo
Wei Jing
Wenxi Liu
Junbo Chen
Jia Pan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Imitation learning is a promising solution to the challenging autonomous urban driving task as experienced human drivers can effortlessly tackle highly complex driving scenarios. Behavior cloning is the most widely applied imitation learning approach in autonomous driving due to its exemption from potentially risky online interactions, but it suffers from the covariate shift issue. To mitigate this problem, we propose a context-conditioned imitation learning approach that learns a policy to map the context state into the ego vehicle's state instead of the typical formulation from both ego and context state to the ego action.  Besides, to make full use of the spatial and temporal relations in the context to infer the ego future states, we design a novel policy network based on the Transformer, whose attention mechanism has demonstrated excellent performance in capturing relations. Finally, during evaluation, a linear quadratic controller is employed to produce smooth planning based on the predicted states from the policy network. Experiments on the real-world large-scale Lyft and nuPlan datasets demonstrate that our method can surpass the state-of-the-art method significantly.


Under review as a conference paper at ICLR 2023
CCIL: CONTEXT-CONDITIONED IMITATION LEARNING FOR URBAN DRIVING
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation learning is promising solution to the challenging autonomous urban driving task experienced human drivers can effortlessly tackle highly complex driving scenarios. Behavior cloning is the most widely applied imitation learn- ing approach in autonomous driving due to its exemption from potentially risky online interactions, but it suffers from the covariate shift issue. To mitigate this problem, we propose a contexl-conditione policy map the context state into the ego vehicle's 's state instead of the typical formulation from both ego and context state to the ego action. Besides, to make full use of the spatial and temporal relations in the context to infer the ego future states, we design a novel policy network based on the Transformer, whose atten- tion mechanism has demonstrated excellent performance in capturing relations. Finally, during evaluation, linear quadratic controller is employed to produce smooth planning based on the predicted states from the policy network. Experi- ments on the real-world large-scale Lyft and nuPlan datasets demonstrate that our method can surpass the trate-of-the-ae method significantly.
imitation learning approach that learns
INTRODUCTION
Planning safe, comfortable, and efficient trajectory in complex urban environment for self- driving vehicle (SDV) is an important and challenging task in autonomous driving Yurttever et al., 2020). Unlike highway driving (Henaff al.] (019), urban driving requires handling more varied road geometry such as roundabouts and intersections while interacting with traffic lights, pedes- trians, and other vehicles. Classic manually-designed rule-based approaches (Fan et al. 2018) have achieved some success in industry but demand tedious human engineering to struggle with diverse real-world cases. Mcanwhile, the rapid development of deep learning techniques motivates researchers (Bojarski et al.) 2016, Pan et al., 2020) to employ deep neural network to model the complicated driving policy. To learn such policy, imitation learning (IL) from human drivers' demonstrations is a promising solution experienced drivers can tackle even extremely challeng- ing situations, and their driving data can be collected at scale.
The simplest II. algorithm is the behavior cloning (BC) method, which has wide applications in autonomous driving (Pomerleau) 1988) [Bojarski et af.) 2016) Codevilla et al.) 2018b) due to its exemption from potentially dangerous online interactions. learns policy in a supervised fashion by minimizing the difference between the learner's action and the expert's 's action in the expert state distribution. However, the BC method suffers from the covariate shift issue (Ross al. 201T), i.e.
the state induced by the learner's 's policy cumulatively deviates from the expert's 's distribution.
To overcome the covariate shift obstacle, existing methods such as DAgger (Ross et al.) 2011 and DART (Laskey et al.] 2017) query supervisor corrections at the learner's 's states or perturbed expert's s states. Since human supervision is hard to collect, recent works like GAIL (Ho & Ermon 2016) seek to provide feedback from a neural network- distribution states generated by the learner's policy. However, these data augmentation methods need either expert supervision or rolling out their policy in the real world or a realistic simulator, which are impractical in autonomous driving. Instead, some researchers attempt to constrain the learned policy formulation to ensure its robustness to the policy error by incorporating control theoretic prior knowledge. For example. Palan et al. (2020); Havens & Hu (202I) pose Kalman or linear matrix inequality constraints on the learned linear policy to guarantee its closed-loop stability in linear
discriminator to recover from out-of-

---

Under review as a conference paper at ICLR 2023
lime-invariant (LTI) system. Yin et al. (2021) relaxes the linear policy formulation into simple
forwafo neural network representation and East (2022) extends the method in Havens & Hu (202T) to polynomial policy and dynamical system. However, the urban driving task is too complex to be handled by these naive policy formulations.
To learn stable and general urban driving policy by imitating only the offline human demonstra- tions, we propose a context-conee context-conditioned imitation learning (CCIL) method, where policy network learned to predict the SDV's future states using only its observed context, different from classic policy taking both ego state and context as input to generate its action (Codevilla et al. 2019 Bansal al. ).19). The motivation is that the ego state can be easily influenced by the policy error, thus leading to the catastrophic distribution shift. On the contrary, the static context elements such as lanes or crosswalks will not influenced by the SDV and the dynamic context element like human drivers will try to recover from its perturbation. Thus, based on the stability assumption of the traf- fie system, we can prove theoretically that our policy formulation can achieve closed-loop stability, thus addressing the distribution shift issue. In practice, as it becomes challenging to accurately plan a trajectory for a SDV without its historical trajectory, we construct our policy network based on Transformer (Vaswani et al. 2017) to exploit spatial and temporal relation information in highly interactive and constrained urban driving scenarios. Furthermore, during evaluation, we employ a linear-quadratic regulator (LQR) (Aström & Murray 2021) yield smooth action based on the SDV's current and predicted states.
The main contributions of this paper can be summarized as follows:
1. To address the covariate shift issue in offline imitation learning. we propose a novel context- conditioned imitation learning method, where policy is learned output the ego state using only its context as input. A robustness assurance is provided for our policy formulation based on an assumption of the context's stability.
2. To apply our method to urban driving, we remove the explicit ego state information input and propose a new ego-perturbed goal-oriented coordinate system reduce the implicit ego information in the coordinate system. Besides, we design a Transformer-base planning network to make full use of the spatial and temporal information in the context.
3. To verify the effectiveness of our approach, we benchmark the real-world large-scale urban driving Lyft (Houston al., (020)) and nuPlan (Caesar et al.] (202T) datasets with state-of-the-ar performance. The video and code can be found at https: //ss//iiieee sess...ae. google. com/view context.condit onedil
2 RELATED WORK
2.1 IMITATION LEARNING FOR AUTONOMOUS DRIVING
The objective of applying IL in autonomous driving is to learn driving behavior mimicking hu- man drivers. The most straightforward solution is BC which minimizes the difference between the learner's and the expert's action on the expert state without demanding extra manually labeled data and online interaction. Early BC applications in autonomous driving such as ALVINN (Pomerleau) 1988) and PilotNet (Bojarski et al. (016) learn an end-to-end policy that directly maps sensor inputs to vehicle control commands using large amount of human driving experience. Recently, Chauf- feurNet (Bansal al., 2019) provides intermediate planning using perception results improve generalization and transparency. However, the BC approach typically leads to the covariate shift between the training distribution and deploying distribution, as minor errors in the policy can lead to deviating from the expert state and then larger errors.
II methods to address the distribution shift challenge can be categorized into online methods and offline model-free and model- based methods. Online methods try to directly match the expert state- action distribution instead of matching the expert state conditioned action distribution like BC. For instance, the method in Zhang & Cho (017) based on DAgger (Ross et al. 2011) queries supervisor actions at the state the learner visits and then adds the new data into the dataset, thus adjusting the expert state distribution to match the leamer'' 's state distribution. To get rid of the requirement of in- teractive expert, several methods like Wang et al. (202T) based on adversarial imitation learning (Ho) & Ermon 2016) utilize discriminator measure the difference between learner and expert's s state-
2

---

Under review as a conference paper at ICLR 2023
action distribution and then compute the reward in reinforcement learning. By raising the policy's 's accumulated reward, the policy's 's state distribution will get close to the expert distribution. How- ever, the reinforcement loop needs online interaction with the environment generate the policy's 's state distribution, which is hard to deploy in safe-critical tasks like autonomous driving. On the contrary, our method matches ego state distribution conditioned on context, which can be learned by supervised learning without interaction with the environment or access to expert supervision.
The most popular model-free IL methods are based on DART (Laskey al., 2017) which avoids the compounding error by providing synthetic examples of how to recover from the deviated state. In Codevilla et al. (2018b), they inject temporally correlated noise into the trajectory to simulate gradual drift away from the desired trajectory. Alternatively, ChauffeurNet (Bansal et al. (019) adds a uniform perturbation to the current SDV state and fits new smooth trajectory that brings the SDV back to the original target location. However, these rule-based trajectory augmentation methods are hard to cover the real motion distribution induced by the learner's i policy and the policy is very likely to learn propensity for perturbed driving. Our method also applies perturbation to the SDV's current position but its role is to blur the ego position information instead of data augmentation. Therefore, our method does not have the trajectory smoothing process during training. Our method is also model-free but we seek to endow the policy with robustness properties by constraining its formulation without resorting recovery examples.
Model-hased IL. methods address the distribution shift by minimizing the difference between a trajectory rolling out in differentiable learned data-driven model with the expert trajectory. PPUU (Amos et al., 2018)) first learn a dynamics model based on variational autoencoder (Kingma & Welling 2013) from data and then train the policy network to output actions that lead to a sim- 1ar trajectory as the expert trajectory. Since the dynamics model is differentiable, the actions can receive gradients from multiple time steps ahead which can penalize actions that will lead to large diverges in the future even if the instantaneous divergence is small. Instead of learning a model, UrbanDriver (Scheel et al. 2022) constructs a differentiable data-driven model using recorded per- ception data and High Definition (HD) maps, where new observations are calculated by coordinate transformation based on the pose of controlled SDV and collected data. However, the performance of the model-basee approach is limited by the model's accuracy.
2.2 IMITATION LEARNING WITH ROBUSTNESS
IL is different from supervised learning by deploying the policy under dynamics, whose robustness considered to be the learned policy's s ability to recover from policy errors. Some researchers have attempted to learn a policy with a stability guarantee using control-theoretic methods by constraining policy and system dynamics. Taylor Series IL (Pfrommer et al., 2022) proves that the trajectory induced by the learner and expert will be close if their derivative difference at expert states is small but computing the high-order derivatives of the expert policy are difficult without sufficient data. Others learn a robust linear (Palan et al. 20200 Havens & Hu I021) or simple feed-forward neural network (Yin et al. 2021) control policy for a linear dynamical system by posing constraints on the policy. Even though the authors in East (2022) extend the robustness guarantee the polynomial system and policy, obtaining guarantees on close-loop stability for the nonlinear autonomous driving system remains challenge. Recently, the CMILe method (Tu et al. 2022)> is capable of training nonlinear policies with the same safety guarantees as the expert but requires online expert access as DAgger. However, our method only constrains our policy in the formulation of receiving context state and producing ego state, whose stability is guaranteed under mild input-to-staie stability assumption on the environment dynamies.
3 THEORETICAL ANALYSIS
We first introduce the notations and definitions used in this paper. For any vector æ E R", | stands for its L norm and || for L norm. For any square matrix A E Raxn, A| denotes its induced L norm and p(A) is its spectral radius (the maximum of the absolute values of its eigenvalues). For every induced matrix norm, we have ((A) ||]|-
3

---

Under review as a conference paper at ICLR 2023
Definition 1 (Comparison Functions) A function Y Rg0 + Rzu is class K. if is continuous, strictly increasing and satisfies ((0) = 0. A function ((x,t) R30 x R30 - Rz0 is class K.C if it is continuous, t) is class K. for each and ((x,) is decreasing for each z.
Definition 2 (input-to-state stability (ISS) (Sontag) 2008)) A discrete-time system Ft+1
f( u Zg = & is input-to- to-state stable if there exists a class K.L.ftuction 3 and a class K. function y such that, for each bounded input u and initial condition & and t 24, it holds that:
(E, uIll & (( t) + 7(122ll)o))
(1)
where = suptez. (ue) is the sup norm of the input. If a system satisfies the ISS property, the distance between any two trajectories must eventually be bounded by its input signal and indepen- dent of initial conditions.
=
We consider performing imitation learning in nonlinear discrete-time system:
xt+1 = f(x,, ut)
(2)
where xt R" is the system state at time f N and u, R" is the control input. For an (8,, c)): an SDV state autonomous driving system, its state can be separated into two parts m which is influenced by the 8; T controlled by the control input u and context state C₁ R SDV state. By viewing the policy error as a disturbance to the system, we apply linearization to the nonlinear system at the expert's state a: to obtain the following linear subsystem:
C++1 Ac + Ba
(3)
with solution:
C+11 = A +
Σ A A
(4)
1=0
Here with abuse of notation, the same symbol Ct. 8t for the context deviation C2 cj and SDV state deviation 8; s
Because the roal-world traffic system without the SDV state deviation Ce+1 Ac, is stable, have (() < 1. For such a Schur matrix, there are constants c > 0 and 0 < a < such that || E ca (Jiang & Wang 2001]. Besides, the context in traffic systems generally includes static map elements that are not influenced by SDV vehicles and intelligent human drivers who can quickly recover from the other vehicle's perturbation. to be input-to-state stable (ISS) with B(r, t) < I r and 7() < Er, where > O. Combining equation I and equation we have:
natural to assume the subsystem in equation
B
B(r,t)= = coo==oorr-- r, 7()) = Σ
co B
< er,
(5)
1=
which implies that c < J. and BB|
<l1o)
Next, we study how the policy in our formulation can be stable under this ISS system. We learn context state feedback control st u, fe(c;) which directly maps the context state to the SDV state. For simplicity, we consider a linear policy u, = Kc., Then, the linear subsystem in equation caan be simplified C++1 = (A + BK)c, whose stable condition is p(A + BK) < 1. Thus, we have if ||| < c(l-ca) then <l-o)
||||+ |B]] co + c(1 o1- 0)
P(A BK) & A + BK|
K
<
(6)
c
This result shows that the stability of closed-loop autonomous driving can be guaranteed as long as the context system is stable enough and the norm of our policy is sufficiently small. In practice, we will add L.2 norm regularization into the training loss of our neural network policy internalize this stability prior.
Finally, we show how the naive BC policy formulation which takes both the SDV state and context state and generates the displacement from the current state can fail with even a fixed context c, = 0. We consider linear policy: u, Kb.8, and update the state 8111 8, + u Then, the dynamics
4

---

Under review as a conference paper at ICLR 2023
is 81+1 (te)st. Because ||-||
cannot obtain the stable condition p(I Kbb) by limiting the bbel as our policy formulation. Thus, the system's stability cannot be guaranteed.
This simple analysis supports choice to overlook the SDV history and current state in the learn- ing to avoid causal confounding and distribution shift. Its advantage will also be validated in our experiments showing that this significantly decrease the off-road rate.
METHOD
As shown in Figure apply our context-conditioned imitation learning method to learn policy for urban driving. The policy network consists of a spatial encoder and temporal encoder. The policy network is trained to minimize the L distance between its predictions and the ground truth trajectories. During the evaluation, we employ LQR controller to generate smooth planning based on the predicted trajectory from the policy network.
Spatial Encoder
Temporal Encoder
OL1-811
o On
Tronsformer Encader
pos.
Cresswalks
Causal Transformer Encoder
Linear
Local Transformer Encoder
$1-200
Predictions
Training
Lanes
Global Transformen Encoder
concat
MLP
Ground truth
Ego Gool
Evaluation
MLP
Planning
Other Agents
Figure 1: Overview of our approach
4.1 INPUT REPRESENTATION
Following Bansal al. (2019). the intermediate representations such oriented bounding boxes from perception system and road networks from HD maps rather than raw sensor data (such as camera images or lidar points clouds)
harnessed improve generalization and interpretability. In detail, there are two types of inputs to the policy network: map context and traffic participants. consider two types map elements, which crosswalks and lanes. Each map element is presented in a polyline which is a sequence of vectors with different associated features Gao al. (0200). The features of crosswalk vectors are their initial and terminal point's position and sequence orders, and lane vectors have additional features such as their traffic light state and lane width. For the traffic participants, we divide them into ego vehicle and other agents. For the ego vehicle, discussed above, we should overlook the features which learned policy to overcome the distribution shift obstacle. Therefore, we only input its mission goal position while removing its history. For each other agent, its features are composed of its type (e.g. car, bicycle, and cyclist), sizes, centroids, and orientations of bounding boxes at several past time steps.
For the map context,
be influenced by the
Even though there is no explicit SDV state in the network input, the observations are typically translated the SDV coordinate system using its rear axle's midpoint as origin and orientation
5

---

Under review as a conference paper at ICLR 2023
x-axis direction in previous works (Bojarski et al. 2016 Scheel et al. 2022). This implicit incorporation of the SDV state can still cause a distribution shift of inputs. To reduce the implicit influence and ensure local observability, whose origin is the SDV position adding a zero mean Gaussian perturbation, and z-axis direction toward its goal position. Note that the perturbation is only required during training and the goal position can be replaced with any point that is not influenced by the driving policy.
adopt ego-perturbed goal-oriented coordinate system
4.2 POLICY NETWORK
The policy network is designed to directly plan trajectory for the SDV using only its context. Since it is more challenging to plan without the historical trajectory, we need to make full use of the spatial-temporal interactions between the SDV and context to infer its future states. Thus, we first embed the observation at each time step into an observation feature by a spatial encoder, and then the observation features in current and previous time steps are embedded generate the predictions by a temporal encoder. We construct our policy network based on the Transformer whose multi-head attention mechanism has demonstrated excellent performance in capturing relations.
To capture the spatial relations between vectors of the same polyline such lane or crosswalk, we employ local Transformer encoder as the vector-vector interaction encoder. Subsequently, we aggregate the features of vectors belonging to the same polyline by max-pooling to obtain polyline-level features. To obtain the agent and goal features, two multi-layer percep- tions (MLP) applied. To model the high-order interactions between the goal, agents, and map elements, we utilize a global Transformer encoder whose goal embedding is output as observation features.
Spatial Encoder:
Temporal Encoder: To take the temporal information and interaction into account, a Transformer encoder with causal self-attention mask is employed embed H historical observation features with a step interval 1. Then, each hidden state of the Transformer is decoded by a full-connected linear layer generate a prediction of the SDV future state at future T time steps. For simplicity, we represent the SDV state with its rear axle's s midpoint coordinate and orientation.
4.3 TRAINING PROCESS
We leverage supervised learning like BC to train the policy network by minimizing the L distance between the predictions and ground truth trajectories, as L is more correlated to driving perfor- mance compared with the mean square error (Codevilla et al., 2018a). To help the network converge and generalize, we additionally introduce an auxiliary task of minimizing the L eтror at all previ- ous time steps and apply a squared L norm regularization to the network parameters O inspired by equation 6 The final loss is:
L=
Σ
||st - 186-66111 +
#-1
Σ h=1
||s
shulla + 1012
A
where is the ground-truth SDV state at time step f and 8%, is the state prediction at the historical time step hT for its future time step. μ an auxiliary hyperparameter and À is a regularization factor.
4.4 EVALUATION PROCESS
Even though the prediction without SDV state input more stable, it has difficulty ensuring the smoothness of the predicted trajectory from the current state. Prior works such as |Vtelli et al. 2002 :: Amos et al. (2018) usually add a differentiable kinematic layer into the policy network to generate physically feasible planning. However, the differentiable kinematic layer will bring in the current SDV state which is undesirable for learning a policy. Therefore. we choose to obtain a smooth trajectory during evaluation by lnnear-uudrrate regulator (LQR) (Aström & Murray (021) which can efficiently minimize the total commutative quadratic cost of a linear dynamic system. We consider finite-horizon, discrete-time linear system whose dynamics is described by:
Pt+1
I
D
2
pt
D
Pr+1
0
1
D
Pt
+
D
(8)
Pr+1
0
P:
I

---

Under review as a conference paper at ICLR 2023
where D is diagonal matrix with the time interval of each step as diagonal entries, (ve, Wt), Pi = (ar, ar), u = (uy, uf) denote the positional and angular velocity, acceleration, and control respectively, subject to quadratic cost function:
J Σ (lp: - 80. 2 + w + wa || 112 + wa + ww [|
where P. is regarded as the planning trajectory and the predicted states š at the current step from the policy network regarded target poses; Was Was Wa and wh, are the weights balance the positional accuracy and the smoothness of the planned trajectory. At the end of the optimization, the LQR will output smooth trajectory for the SDV to follow.
EXPERIMENTS
5.1 DATASET
To benchmark method's performance, we conduct experiments datasets: Lyft Level 5 Prediction Dataset (Houston et al. 2020): contains about 1000h urban driving demonstrations Palo Alto, which have been separated into independent scenes of nearly 25s at 10Hz. We train our network on the provided 100h (16265 scenes) subset Scheel et al. (2022) and test with all 16220 validation scenes. nuPlan Dataset (Caesar al.) 2021) provides an urban driving dataset with 13 13h2h of human driving data from 4 cities (Boston, Pittsburgh, Las Vegas, and Singapore). Due to the huge difference in traffic rules and patterns different cities, extract driving data in Las Vegas as Phan-Minh et al. (2022). Then, separate the data into independent scenes of 25s at 10Hz like Lyft and filter out the scene without a mission goal. After filtering, we obtain 63181 training, 4774 validation, and 6386 testing scenes. More details about the data prepossessing and corresponding model details are presented in the appendix.
real-world large-scale
5.2 CLOSED-LooD EVALUATION
To evaluate the closed-loop performance of our method, we use a log-replay simulator as prior work (Scheel et al. 2022 Huang et al. 2022). At each step in the log-replay simulator, the SDV vehicle updates its state according to the planned trajectory, while the other agents are assumed to follow their recorded trajectories in the dataset. For both datasets, we evaluate our method for 25s at 10Hz with the following metrics:
Collision Rate: If the SDV collides with the other agents at any time step scene, the scene is deemed as collision scene. The collision rate is the ratio of the collision scenes in all scenes.
Off-road Rate: In nuPlan, we use the official drivable area compliance metric, i.e. the distance of corner of SDV's bounding box from the drivable area is more than 0.3m. But in Lyft, without access the drivable area, the SDV deviates laterally from the human driver's ground truth more than 2m like in a scene, the scene will be deemed off-road as Scheel et al. (2022).
we count the rate of the absolute value of the acceleration more than 3 m/s² over time steps to quantify the comfort and feasibility of the planned trajectory.
Discomfort:
L2: we use the average 1.2 position errors between the roll-out trajectory and the human driver's ground truth quantify the human driving similarity.
5.3 PERFORMANCE EVALUATION
On the Lyft dataset, compare methods against three state-of-the-art methods demonstrate the advantage of the proposed framework:
Raster-perturb: official BC planning baseline (Houston et al. 2020), which based on ResNet50 receives a Bird-Eye- View (BEV) representation of the scene surrounding the SDV and produces trajectory of position and yaw displacements. To augment data, perturbation is applied to the current SDV position, and then new kinematically feasible trajectory to reach the original endpoint is generated as ChauffeurNet.

---

Under review as a conference paper at ICLR 2023
Table I: Comparison with baselines of 25s closed-loop performance Lyft and nuPlan dataset.
There are no variances in Raster-perturb BC-perturb, and UrbanDriver because we evaluate the deterministic pre-trained models in deterministic simulator.
Model
Num params
Collision( %)
Off-road ())
Discomfort( %))
L2(m)
Raster-perturb
23.6M
15.48
5.06
4.00
5,90
BC-perturb
1.8M
9.38
6.77
39.10
4.77
UrivnDriver
1.8M
13.28
7.27
39.41
5.74
TD3+BC
2.8M
22.53=1.76
15.21=0.97
4.86=0.47
6.34+0..41
Vector- Chauf-Cu
1.5M
10.12=0.23
3.40=#0.32
5.42::0.44
5.03+40:43
CCTL (ours)
1.5M
3.32+L0.15
0.62++0.13
4.33+t0.22
1.23 + 0.08
Raster
23.6M
62.45=3.45
32.15
21.50=3.19
22.71#2.20
LaneGCN-perturb
2.0M
60.63+2.34
34.25+1.65
17.26+1.80
21.21+1.81
TD3+BC
2.8M
39.12=2.21
18.59+1.04
10.56+00.55
15.04+t1.62
Vector-Chauffeur
1.5M
1..1
10.11
12.53+1.17
6.12.10.87
CCIL (ours)
1.5M
6.91
3.08
1.16:10.05
3.68
BC-perturb: BC model provided by |Schee et al. (2022) with the same trajectory perturbation and output as Raster-perturb but its inputs are presented in vector formulation, which are processed by a VectorNet (Gao et al. 2020) and Transformer.
UrbanDriver: offline policy gradient method proposed by (Scheel et al. (2022) to imitate the ex- pert's policy exploiting differentiable data-driven simulator with the same data and model structure BC- BC-perturb.
On the nuPlan dataset, we compare our methods against two official baselines learned by BC due to a lack of other prior works:
Raster: a raster-based model that uses the ResNet-50 backbone to encode SDV, agent, and map information raster layers plan the SDV's trajectory.
vector-based model that uses a series of MLPs to encode SDV and agent signals and one LaneGCN (Liang al. 2020) to encode vector-map elements and fusion network to capture lane and agent trajectories perturbed and other agents randomly dropped out.
LaneGCN-perturl)
through attention layers. To augment data, the SDV
In addition to these models provided by prior works, we also learn our model by a representative method in behavior cloning and offline reinforcement learning on both datasets:
TD3+BC (Fujimoto & Gu 2021): adds behavior cloning term to the policy updating of Twin Delayed Deep Deterministie Policy Gradient (TD3) (Fujimoto et al.) 2018) for implicit policy straint. We construct the offline RL dataset by applying the trajectory perturbation augmentation and consider the collision and comfort for the reward design.
Cector-Chauffeur: our model learned using the same data augmentation method as ChauffeurNet including trajectory perturbation and ego past motion dropout. And represent the data in the same coordinate system which uses the ego location as the origin and its heading perturbed by uniform noise the orientation.
The performance is shown in Table It demonstrates that our method can outperform previous work significantly on both datasets. The collision rate on the nuPlan dataset is higher than the Lyft dataset because nuPlan's scenarios are more complex with more road agents.
5.4
ABLATION STUDY
The following ablation experiments on the Lyft dataset are used to expose the significance of differ- components of our model, whose results are shown in Table
8

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation experiments 25s closed-loop performance Lyft dataset
Model
Perturb
Ego
Collision(%% (%)
Off-road %)
Discomfort %
L2(m)
explicit ego
20.29++0.88
19.18+22.98
0.57+00.15
5.50+=0.55
ego dropout
14.05+1.55
5.02++0.88
0.63++020
4.15+0.47
ego coordinate
11.31
9.79+1134
0.95++0.05
3.87=0.11
std=0
0.05
2.86++0.25
0.89+10100
3.46+1034
std=1
3.39:t.11
1.00=0.16
1.99:#0.15
2.10=0.05
std=2 (ours)
3.
0.62:10113
4.33:10.22
1.23
std=3
3.42::0.12
0.49
7.35
0.91
w/o causal Trans
4.28:
1.43
6.53+L0.26
1.63+
w/o LQR
3.81+0.14
2.07++0.12
89.05+
1.02+0002
w/o regularization w/o auxiliary
V
4.0740.16
1.05#0.14
4.96:10.30
1.23=0.09
4.56++0.29
1.02+0.06
3.23++0.31
1.92+00007
Network input: We first study the importance of removing explicit ego information from the net- work input. We consider two ways to bring the explicit ego input back. The first one is to directly input ego past positions and then process similarly the goal position. The other one tionally introduce a dropout of 50% at the ego input during training as Bansal et al. (2019). We can observe that both explicit ego information input ways, there a steep drop the collision and off-road rate and L. distance due to the covariate shift issue.
addi-
Coordinate system: analyze the impact of the implicit ego information the coordinate system, first consider replacing ego-perturbed goal-oriented coordinate system with the ego-centric coordinate system ChauffeurNet using orientation uniformly around the heading. We observe that the ChauffeurNet coordinate system leads to inferior closed-loop performance. In our coordi- nate system, we add Gaussian noise with zero mean to the ego current position to obtain the origin. Thus,
increase the Gaussian noise's s standard deviation to diminish the implicit ego informa- tion. We can observe that with the increasing of the standard deviation, the discomfort increases but the off-road rate and closed-loop 1.2 decreases, which implies that the implicit ego information may improve the instantancous prediction accuracy but deteriorate the closed-loop performance.
To demonstrate the importance of the temporal information and the effectiveness of the causal Transformer in capturing the temporal interactions, we replace it with a MLP. We find that the causal Transformer can achieve better performance. Besides, to show the importance of the LQR module in enhancing comfort, we ablate it. Removing the LQR mechanism leads to huge drop comfort.
Architecture:
Loss: To demonstrate the effectiveness of the additional term, we remove the auxiliary and regular- ization loss term separately. The result in Table shows that they are both beneficial to improving performance.
CONCLUSION
We have proposed new offline imitation learning method mitigate the distribution shift of behav- ior cloning, where we learn neural network to predict the SDV's future positions and orientations without SDV information by making full use of its constraints from and interaction with the context. Firstly, we analyze theoretically the stability of our policy formulation. In addition to removing the explicit SDV information input to the network, we present a new SDV-perturbed goal-oriented coordinate system for representing represent the observation input to remove the implicit SDV in- formation and ensure local observability. We design Transformer- based network to make full use of the history information to handle the more challenging learning task. Finally, demonstrate the effectiveness of our approach in two real- world large-scale datasets with of perfor- mance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable mpe for end-to-end planning and control. NeurlPS, 31, 2018.
Karl Johan Aström and Richard M Murray. Feedback systems: an introduction for scientists and engineers. Princeton University Press, 2021.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In RSS, 2019.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.0336,, 2016.
Holger Caesar, Juraj Kabzan, Kok Scang Tan, Whye Kit Fong, Eric Wolff, Alex Lang. Luke Fletcher, Oscar Beijbom, and Sammy Omari. Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop, 2021.
Felipe Codevilla, Antonio M Lopez, Vladlen Koltun, and Alexey Dosovitskiy. On offline evaluation of vision-based driving models. In ECCV, pp. 236-251, 2018a.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. In ICRA, pp. 4693-4700, 2018)
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In CVPR, pp. 9329-9338, 2019.
Sebastian East. Imitation learning of stabilizing policies for nonlinear systems. European Journal of Control, pp. 100678, 2022.
Haoyang Fan, Fan Zhu, Changchun Liu, Liangliang Zhang, Li Zhuang, Dong Li, Weicheng Zhu, Jiangtao Hu, Hongye Li, and Qi Kong. Baidu apollo em motion planner. arXiv preprint arXiv: veXivv1807.00m 80.008048 2018.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 1-20012--20145 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 10-15 Jul 2018. URL /
ppressv80/ fu motool Ba. html
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid, Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR. pp. 11525-11533 2020.
Aaron Havens and Bin Hu. imitation learning of linear control policies: Enforcing stability and robustness constraints via Imi conditions. In ACC, pp. 882-887, 2021.
Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncer- tainty regularization for driving in dense traffic. In ICLR, 2019.
Jonathan Ho and Stefano Ermon, Generative adversarial imitation learning. NeurIPS, 29, 2016.
John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and hours: Self-driving motion pre- diction dataset. In CoRI,, 2020.
Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. Differentiable integrated motion pre- diction and planning with learnable cost function for autonomous driving. arXiv preprint arXiv:2207. 10422, 2022.
10

---

Under review as a conference paper at ICLR 2023
Zhong-Ping Jiang and Yuan Wang. Input-to-state stability for discrete-time nonlinear systems. Au- tomatica, 37(6):857-869, 2001.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes. arXiv preprint
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In CoRL, pp. 143- PMLR, 2017.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In ECCV, pp. 541-556, 2020.
Malayandi Palan, Shane Barratt, Alex McCauley, Dorsa Sadigh, Vikas Sindhwani, and Stephen Boyd. Fitting a linear control policy to demonstrations with a kalman constraint. In Learning for Dynamics and Control, pp. 374-383. PMLR, 2020.
Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntack Lec, Xinyan Yan, Evangelos A Theodorou, and Byron Boots. Imitation learning for agile autonomous driving. The International Journal of Roboties Research, (9(223)2286--333 2020.
Daniel Pfrommer, Thomas TCK Zhang, Stephen Tu, and Nikolai Matni. Tasil: Taylor series imita- tion learning. arXiv preprint arXiv:2205. 148 2022.
Tung Phan-Minh, Forbes Howington, Ting-Sheng Chu, Sang Uk Lee, Momchil S Tomov, Nanxiang Li, Caglayan Dicle, Samuel Findler, Francisco Suarez-Ruiz, Robert Beaudoin, et al. Driving in real life with inverse reinforcement learning. arXiv preprint arXiv:2206.03004 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. NeurlPS, 1, 1988.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In AISTATS, pp. 627-635, 2011.
Oliver Scheel, Luca Bergamini, Maciej Wolczyk, Blažej Osiński, and Peter Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In CoRL, pp. 718-728, 2022.
Eduardo Sontag, Input state stability: Basic concepts and results. In Nonlinear and optimal control theory, pp. 163-220. Springer, 2008.
Stephen Tu, Alexander Robey, Tingnan Zhang, and Nikolai Matni. On the sample complexity of stability constrained imitation learning. In Learning for Dynamics and Control Conference, pp- 180-191. PMLR, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017.
Matt Vitelli, Yan Chang, Yawei Ye, Ana Ferreira, Maciej Woiczyk, Blažej Osiński, Moritz Niendorf, Hugo Grimmett, Qiangui Huang, Ashesh Jain, al. Safetynet: Safe planning for real-world self- driving vehicles using machine- policies. In ICRA, pp. 897-904, 2022.
Pin Wang, Dapeng Liu, Jiayu Chen, Hanhan Li, and Ching- Yao Chan. Decision making for au- tonomous driving via augmented adversarial inverse reinforcement learning. In ICRA, pp. 1036- 1042. IEEE, 2021.
He Yin, Peter Seiler, Ming Jin, and Murat Areak. Imitation learning with stability and safety guar- antees. IEEE Control Systems Letters, 6:409-414, 2021.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-5846, 2020.
Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simulated driv- ing. In AAAI, pp. 2891-2897, 2017.
11

---

Under review as a conference paper at ICLR 2023
A MAP PREPOSSESSING
In Lyft, there are only two types of map elements: lane and crosswalk. In nuPlan, the map elements consist of a lane, lane connector, intersection, stop line, crosswalk, walkway, and car park. We categorize them into polyline elements including lanes and lane connectors, and polygon elements including stop lines, crosswalks, intersections, walkways, and car parks.
For each polyline element, is approximated by sequence of vectors with interval of 3 meters. For both datasets, we prepossess the polyline map into graph to take advantage of the topological information. We connect each vector with its nearest left, right, and next vector if exists. The nearest left or right vector means the nearest vector of its reachable left or right polyline. And the distance between vectors is represented by the Euclidean distance between their starting points. After building the vector graph, we can compute the travel distance between any two vectors using Dijkstra's 's algorithm. For both datasets, the polyline vector has shared features including coordinates of starting and ending point, distance to left and right vectors, distance to the mission goal, traffic light states (red or green), and the sequence order. However, lane vectors in Lyft have a lane width feature because the vectors approximate computed middle lines of lanes, while the lane lane connector vectors nuPlan have a lane left and width feature because the vectors approximate an annotated baseline. In addition, the polyline vectors in nuPlan have additional speed limit and type features.
For each polygon element, we approximate it also with sequence of vectors. In the Lyft dataset, the crosswalk vectors are directly constructed by connecting the original sequential annotation coordi- nates. But approximate each polygon with fixed 20 vectors because the annotation point number too large for elements like intersection. For both datasets, each polygon vector has features in- cluding coordinates of starting and ending points, and its order sequence, while vectors in nuPlan have additional type features.
The inputs to our neural network are composed of two types of map elements: polyline and polygon, and two types of agent elements: other agents and ego goal. The missing inputs are padded with zeros and masked out when calculating the attention. The origin is the perturbed SDV position.
30 topologically nearest polylines with vectors whose starting points are within 35m from the origin. The topological distance between the origin and a polyline is the minimum of topological distances between the origin and its vectors.
Polyline:
Polygons:
20 polygons whose boundaries are within 35m from the origin. If there are more than 20 polygons, they are selected according to the importance of their types: stop line, crosswalk, intersection, walkway, and car park.
the nearest 30 agents whose oriented boxes' centroids are within 50m from the origin. The agent features in Lyft include its centroid coordinates, yaws, shapes, types, and relative times in the past 2 and current steps. The nuPlan agents additionally have velocity features as they are provided.
Agents:
the x, coordinates of the SDV's mission goal of scene. In Lyft, the mission goal is not provided, so regard the ending point of the lane where the SDV locates at the last time step of the scene as the mission goal. In nuPlan, we directly take the provided mission goal at the last time step of the scene as the scene mission goal.
Goal:
B MODEL
For both datasets. the same model architecture is used, whose hyper- parameersss are listed in Table
C TRAINING
Our model is trained using the Adam optimizer with a learning rate of 0.0005, 10000 steps linear warm-up, 3 = (0.9. 0.999). and batch size 128. We stop training after 30 epochs and select the model with smallest validation collision rate for evaluation. We train all models except the pre- trained model in Lyft dataset independently for times and then report the mean and std of their performances.
12

---

Under review as a conference paper at ICLR 2023
Table 3: Hyper-parameters for both datasets
Hyper-parameter
Value
Future steps T
15
All Transformers dropout
0.1
All Transformers head number
8
All Transformers hidden size
128
Local Transformer layer number
3
Global Transformer layer number
Causal Transformer layer number
3
Causal Transformer length H
15
Causal Transformer interval I Auxiliary weight μ
2
0.3
Regularization weight A
0.0001
LQR angular velocity weight who
0.1
LQR acceleration weight wa LQR angular acceleration weight wa LQR jerk weight wh
0.1
0.1
0.1
D EVALUATION
For Lyft baselines, we directly evaluate the pretrained model provided by the Lyft dataset and UrvanDriver. The Raster-perturb model trained train.zarr for epochs at https: com / http:////pbb/ntteeeeee/ap//pp//a The BC-perturb and UrbanDriver are from Open Loop and Urban
plamiagg/train.
Driver in urban_drriver/train..f
We evaluate all Lyft models from the second time step to compute current velocity using position information as input to the LQR with zero assumed initial acceleration.
For nuPlan baselines, we train the official models by ourselves using provided hyper-paramene and evaluate from the first time step because the velocity and acceleration information provided.
E RUNTIME
We conduct runtime experiments using a single Nvidia GeForce GTX 1080 GPU and an Intel 8700@3.22 3.2GZz CPU Lyft dataset. We measure the runtime of each method its and std over all time-steps in an evaluation. The runtime results shown in Table(4connsider all components each model including tetta-prepossesinng. model inference and control. We observe that our architecture can achieve higher data processing efficiency and medium model inference efficiency compared with other methods. Our method takes longer total execution time due the extra LQR control module which does not exist in prior works because they only focus on optimizing positional accuracy but comfort. However, our approach can still be executed in real-time on this hardware.
Table 4: Averaged runtime per frame of individual components for each method
Model
Data process (ms)
Model inference (ms) Control (ms)
Total (ms)
Raster-perturb
6.03:10.61
4.62-10.16
10.6510.64
BC-perture
6.69:10.72
6.78+55.26
13.47=5.665
UrbanDriver
6.33+1141
12.78+8.80
19.11-9.23
CCIL
4.92-10.63
6.74+L0.34
11.09-+0.27
22.75
13

---

Under review as a conference paper at ICLR 2023
F TOY EXPERIMENT
We design a toy experiment to vividly show our method's ability to reduce compounding error. In the experiment, we use synthetic data from very ideal and simplified scenario where SDV moves under ring road network with fixed m/s at IHz, as shown in Figure 2 During training, the radius of the ring is a variable with range from 10m 100m and the circular lane are represented series of fixed lane points with same interval of nearly Im.
radius U[10,100]
1m
Figure 2: Ring road
We compare our CCIL methods with several baseline methods introduced above including BC, BC- perturb, UrbanDriver. The inputs of these baseline methods are composed of two parts: SDV state (its position and orientation in the past 10 time step) and context (nearest 10 lane points) in the ego coordinate system, while our methods only takes the context the past 10 time step as inputs in the ego-perturbed center-oriented coordinate system. The ego-perturbed origin coordinate system means using the SDV position added a zero-mean one-std Gaussian perturbation as the origin and orientation to the center of the circular road the x-axis direction. The trajectory perturbation is applied to augment the data in the BC-perturb method. For outputs, the BC and CCIL method generate the relative position and yaw at the next time step and the BC-perturb method produces the next time steps. In the UrbanDriver method, we unroll the policy for 32 time steps.
In each methods, employ two-layer MLP with hidden size of 128 train the neural networks using Adam optimizer with a learning rate of 0.0001 with random initial weights for 100 times. We stop training after 10000 steps and then unroll the policy from one random starting point on the circle of radius 50m for 100 time steps. The 100 closed-loop trajectories for each method are depicted in Figure 3 We can observe that some trajectories in the baseline methods deviate from the road due to the covariate shift issue but the trajectories in our method keep following the route.
policy network. We
14

---

Under review as a conference paper at ICLR 2023
BC
Pe-turb
75
50
25
-25
-50
UrbanDriver
CCIL
75
25
-25
-75
-75
-50
-25
25
50
75
-75
-50
-25
25
50
Figure Closed-loop trajectories of each model trained on the toy dataset.
15


Reviews:
[{"review_id": "w1Vy11EIe6K", "reviewer": "Reviewer_VSrE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an offline method to overcome the covariate shift issue in imitation learning. A context-conditioned imitation learning method was proposed, which learns a policy to map context state into  ego vehicle's state without any history ego vehicle's information. To apply the method to urban driving, an ego-perturbed goal-oriented coordinate system was implemented, which helps to reduce the implicit ego information. Finally, model performance was tested on Lyft and nuPlan datasets with several baselines, which demonstrates proposed method can outperform all baselines. ", "summary_of_the_review": "Though there exist some problems in Figure and Benchmark, this paper is still considered as full of novelty and beneficial to the field. The paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "Strength:\nThis paper proposes a novel offline learning method and a new coordinate system representation, which aims to mitigate the covariate shift problem. The proposed model outperforms the state of the arts in both Lyft and  nuPlan datasets in multiple matrics including collision, off-road, discomfort and L2 position errors with ground truth. In addition, the video and code are released, which will benefit the research of imitation learning. Besides, the paper is well written and has clear framework. In summary, the strength are novelty, performance and opensource.\n\nWeaknesses:\n1. Figure\nFigure1:\nFigure1 is not elegant, impressive or fresh. The authors really need to pay more attention to Figure1 and spend much more time on it.\n\nFigure2:\nI do not understand which method the authors choosed as \"BC\" method in figure2. The authors should explain more about Figure2 in details.\nI do not understand why only one \"BC\" method was selected as comparison with proposed method in toy experiment either.\nIn Performance Evaluation the authors compared with three methods including Raster-perturb, BC-perturb and UrbanDriver. At least it is required to show the closed-loop trajectories of all these models.\n\n2. Benchmark\nBenchmark should take the state of the art offline reinforcement learning algorithms such as Conservative Q-Learning(CQL) into consideration, which makes the result more convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "u-Htavhq1Bk", "reviewer": "Reviewer_cGNi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents CCIL, a novel behavior cloning approach that learns a driving policy from human demonstrations. It adopts an ego-centric scene representation with a transformer-based architecture. The network predicts a sequence of future ego SDV states, which are converted to control actuations by a LQR. CCIL is evaluated in closed-loop planning driving benchmarks Lyft and nuPlan and attains state-of-the-art performances on both benchmarks.\n", "summary_of_the_review": "The paper presents CCIL, a behavior cloning approach for driving policy learning. Although the presented approach is effective and attains state-of-the-art performance on both the Lyft and nuPlan benchmarks, aspects/main claims of the paper are not novel and heavily explored in driving policy learning/autonomous driving. Hence I do not recommend acceptance of the paper at the moment.", "strength_and_weaknesses": "### Strengths\n- The paper is well written and the presentation is clear.\n- The proposed method is effective and achieves state-of-the-art performance. This is the strongest aspect of this paper in my opinion.\n\n### Weaknesses\n- I am not sure how helpful the theoretical analysis is. The assumptions in equation 3 and 6 which linearizes the state and policy does not match the real world or actual implementation of methods. I am also curious why the example at the end of section 3 particularly chooses the formulation of “u_t=s_t + K_bc s_t”.\n- I am afraid that the proposed method has limited novelty. Using an ego oriented coordinate space for observation and predicting future states has been explored in behavior cloning for driving policy learning before [1,2] and is largely a standard practice in motion forecasting [3,4] (although the latter also contains ego state information as opposed to CCIL). \n- I would like to see a latency analysis of the whole system as well as individual components of the methods, and how it compares with the baseline methods.\n- I would like to see an ablation where the causal transformer encoder is trained to predict only the current step’s state predictions instead of all H steps.\n\n[1] ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, Bansal et al.\n\n[2] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving, Chitta et al.\n\n[3] MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction, Chai et al.\n\n[4] Diverse Multi-Future Prediction and Planning for Self-Driving, Cui et al.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "quB826BhZm8", "reviewer": "Reviewer_oTYL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method based on the Past Motion Dropout hyperparameter from ChaufferNet, the behavioral cloning approach to autonomous vehicle motion planning. With the same motivation as ChaufferNet authors, they suggest to set the dropout parameter to 100%, such that neural network policy does not have access to past ego states. Similarly to ChaufferNet, they implement initial state perturbations as well. To compensate for the lack of temporal smoothness, the authors add a linear quadratic regulator. The authors also adopt a common Transformer-based architecture.", "summary_of_the_review": "My suggestion is borderline due to minor novelty compared to ChaufferNet and the lack of real world testing. It is not clear how many parameters the proposed approach and selected baselines have, and if the results are statistically significant.", "strength_and_weaknesses": "Strengths:\n- the paper is well written and is easy to follow\n\n- it is great that the approach is evaluated on a publicly available planning benchmark\n\nWeaknesses:\n- there is no statistical significance evaluation of both the main comparison table and the ablation table. Planning benchmarks are known to have high variance, and it is not clear how training and evaluation randomness affect the results. I suggest the authors follow reinforcement learning experiments best practices, e.g. [1]\n\n- baseline selection: it is explained how the model architectures differ and how many parameters they have. Would be great to add a column with numbers of parameters for every model in Table 1, for example:\n\n|Benchmark|Model | Num params | Comment |\n|:--|:--|:--:|:---:|\n|Lyft|Raster-perturb | 25M | based on ResNet-50 |\n|Lyft |BC-perturb | 2M | based on github colab |\n|Lyft | UrbanDriver | 2M | or 3.5M in the paper, not clear |\n|Lyft | CCIL | ? | ? |\n|nuPlan|Raster | 25M | based on ResNet-50 |\n| nuPlan |Vector-perturb | ? | |\n| nuPlan |LaneGCN-perturb | ? | |\n| nuPlan | CCIL | ? |  |\n\nBased on the open-source code of UrbanDriver, it looks like BC-perturb and UrbanDriver models are small compared to ResNet-50 based Raster-perturb. Depending on the number of parameters in the proposed method, these approaches could be not directly comparable.\n\n[1] Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ds6YNj6tpo", "reviewer": "Reviewer_oLbQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work builds a pipeline for closed-loop planning via behavior cloning.\nTheir propose to use only past local context (as opposed to SDV state and context in prior work), and use a transformer to predict future states, which is then turned to actions via LQR controller.\nThey show significant improvements over prior methods on the nuPlan and Lyft closed loop planning benchmark. ", "summary_of_the_review": "This work introduces a fairly counter-intuitive but effective approach and show significant improvements on closed-loop planning. However, I believe the paper does not have the appropriate amount of polish for the venue.", "strength_and_weaknesses": "### Strengths\n\n* Experiments show clear improvements over IDM, FLOW, COPO\n* Overall mathematical notion is for the most part clear and well-motivated\n\n### Weaknesses\n\n* The main motivation seems quite counter-intuitive at first glance. Taking out ego-state seems like it would harm performance.\n* Overall writing could be improved - several sections read off like lists.\n* Figures could use some polish and better descriptions.\n* nuPlan baselines are **extremely** weak which raises some concern.\n\n### Minor Issues / Questions\n\n* In table 2: it looks like w/o regularization is quite strong.\n* How does the method perform without the LQR controller?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject