DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles
Authors: Peter Karkus
Boris Ivanovic
Shie Mannor
Marco Pavone
Venue: CoRL 2022 Poster (2022)
Keywords: Autonomous Vehicles
Differentiable Algorithms
Motion Forecasting
Planning
Control

Abstract: Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, prediction, planning, control, etc. While modularity improves reusability, interpretability, and generalizability, it also suffers from compounding errors, information bottlenecks, and integration challenges. To overcome these challenges, a prominent approach is to convert the AV stack into an end-to-end neural network and train it with data. While such approaches have achieved impressive results, they typically lack interpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks. To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, a differentiable and modular stack for prediction, planning, and control. Crucially, our model-based planning and control algorithms leverage recent advancements in differentiable optimization to produce gradients, enabling optimization of upstream components, such as prediction, via backpropagation through planning and control. Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect planning. Beyond these immediate benefits, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures.

DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles
Peter Karkus', Boris Ivanovic', Shie Mannor Marco Pavone
'NVIDIA Research, Technion, SStanford University (pkarkus, bivanovic, smannor,
(ypavone)@nvidee vrrnnor,
com
Abstract: Autonomous vehicle (AV) stacks are typically built in modular fash- ion, with explicit components performing detection, tracking, prediction, planning, control, etc. While modularity improves reusability, interpretability, and general- izability, also suffers from compounding errors, information bottlenecks, and integration challenges. To overcome these challenges, prominent approach is convert the AV stack into an end-to-end neural network and train it with data. While such approaches have achieved impressive results, they typically lack in- terpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks. To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, differentiable and modular stack for prediction, planning, and control. Crucially, our model-based planning and control algorithms leverage recent advancements produce gradients, enabling optimization of up-
in differentiable optimization stream components, such as prediction, via backpropagation through planning and control. Our results the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect plan- ning. Beyond these immediate benefits, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures. Project website: https://sites google. cwm/vicw/ difee//iffaaacckk
Keywords: Differentiable Algorithms, Autonomous Driving, Planning, Control.
Introduction
Intelligent robotic systems, such as autonomous vehicles (AVs), are typically architected in modular fashion and comprised of modules performing detection, tracking. prediction, planning, and control, among others [1, 2, 3, 4, 5. 6, 7, 8]. Modular architectures are generally desirable because of their verifiability, interpretability and generalization performance; however, they also suffer from compounding errors, information bottlenecks, and integration challenges.
promising line of work tackling these issues focuses on making AV stacks more integrated (by relaxing inter-module interfaces) and data-driven (by optimizing modules jointly with respect to their downstream task). For example, in the context of AV perception, recent work has achieved substantial performance gains by jointly training tracking models with detection [9] and prediction models [10, 11]. To extend such a joint, data-driven approach decision making, existing approaches replace hand-engineered components, e.g., planning and control algorithms, with deep neural networks [12, 13, 14]. As neural networks differentiable, they can be optimized end-to-end for a final control objective; however, they offer weaker generalization, little to no interpretability or safety guarantees.
We introduce DiffStack, a differentiable AV stack with modules for prediction, planning, and control that combines the benefits of modular and data-driven architectures (Fig. 1). The prediction module in DiffStack is a learned neural network that predicts the future motion agents; the planning and control modules are principled, hand-engineered algorithms that produce AV actions given the current world state and motion predictions. Importantly, our hand-engineen engiineeiee planning and control algorithms are differentiable, enabling the training of the upstream prediction module for downstream control objective by backpropagating gradients through the algorithms. In doing so, DiffStack can jointly
6th Conference on Robot Learning (CoRL. 2022), Auckland, New Zcaland.

---

Standard stack
DiffStack
End-to-end policy
Perception
Prediction
Planning
Control
Perceptiion
Prediction
Planningg
Control
0000
Modular
Differentiable
Differentiable
Figure 1: We desire AV architectures that are modular and differentiable, combining interpretability. reusability, and generalizability with data-driven end-to-end training. We take an important step towards this end with DiffStack, differentiable AV stack for prediction, planning, and control.
optimize the entire stack for the final control objective, as in end-to-end neural networks, however it also maintains the interpretability and formal guarantees of standard modular AV architectures.
To make the planner and controller differentiable, we build upon the growing body of work on differentiable algorithm networks [15]. In particular, we leverage the diffcrentiable Model Predictive Control (MPC) algorithm [16] for our controller. While differentiable algorithms show great promise, most prior works consider relatively simple control problems, e.g., pendulum and cartpole in [16]. To the best of our knowledge, this is the first work demonstrate that, through careful design choices and integration, differentiable algorithms can be composed into stacks and trained with real-world data for AV prediction, planning, and control.
We evaluate DiffStack both open-loop and closed-loop simulation settings using the large-scale, real-world nuScenes dataset [17]. Our results show some immediate benefits of differentiable stacks: by training prediction model with respect to the final control objective, DiffStack increases the effectiveness of predictions for decision making by up to 15% over a large number of diverse scenarios. DiftStaack achieves this by, e.g., learning to make fewer prediction errors that would negatively affect planning. Further, DiffStack outperforms alternative handcrafted planning aware prediction losses; compensates for artificially introduced integration errors; and tunes an interpretable cost function make resulting plans demonstrates the feasibility and potential of differentiable AV decision making stacks, making an important step towards class of data-driven yet modular AV architectures.
similar to human driving. Beyond these immediate benefits, our work
Related work
Modular AV architectures. Many state-of-the-art methods for AV perception have achieved substantial gains by relaxing bottlenecks between learned neural network modules and training multiple modules together. Examples include works that more tightly integrate object detection and tracking [9], as well as prediction with object detection [11], classification [18], and tracking [19, 20, 10]. While successful, these works focus AV modules where neural networks already commonly employed and thus differeentablee, i.c., perception and prediction. Looking downstream, there are many works that focus on more tightly integrating prediction within planning [21, 22, 23, 24, 25], however, these works only use predictions planning rather than improve prediction module with information from planning. Recently, works such as [26, 27, 28] present potential avenues for designing upstream modules in a planning-aware fashion. In particular, Ivanovic and Pavone [27] propose an evaluation metric for motion prediction that considers the impact of predictions on downstream planning. While this metric can be used to train prediction with information from planning, we will show Section that the resulting improvement is less than with DiffStack.
Data-driven AV architectures. Another line work eschews modularity entirely, and learns a policy network directly from data [12, 29, 30]. The approach benefits from the removal of information bottlenecks and performance scaling with increasing dataset sizes. However, fully end-to-end approaches lack interpretability, important for debugging, verification, and safety guarantees; it is also unclear how well purely data-driven methods can generalize to unseen or rare-but-critical scenarios. Accordingly. recent works have focused on reintroducing structure and interpretability to learned policy networks. Most notably, the Neural Motion Planner [13] and follow-up methods [31, 32, 14] use object detection and agent behavior prediction as interpretable intermediate auxiliary objectives while training joint backbone network from sensors directly to decision making. An important

---

plan
VC
V
(-8:3) PecA Past tracklets
anguis (iead) Hrrt
(1:T)) Fetr ( (LTT)
Teti Sea laffgi)))
Ego trajectory
Prediction Model
Differentiable Planner
Differentiable Controller
Figure 2: DiffStack is comprised of differentiable modules for prediction, planning, and control. Importantly, this means that gradients can propagate backwards all the way from the final planning objective, allowing upstream predictions to be optimized with respect to downstream decision making.
downside of this approach that all components are neural networks, preventing the incorporation of principled planning and control algorithms, which are otherwise common in practice.
Differentiable algorithms. The idea of making existing model-based algorithms differentiable opens up new design paradigms that blend data-driven deep Icarning with classic model-based algorithms [15]. When both neural network models and fixed algorithmic computation can be expressed in the same "neural programming" framework, one can perform joint end-to-end learning even in structured modular architectures. Differentiable algorithms have been explored in various robotic domains including localization [33, 34], mapping [35, 36), navigation [37, 38, 15], and robot control [39, 16, 40, 41]. Our work is motivated by the initial successes of these approaches in rather simple robotic domains. To the best of our knowledge, this is the first work to apply differentiable algorithms in the context of autonomous driving stacks including prediction, planning, and control.
3 DiffStack: A Differentiable and Modular Autonomous Vehicle Stack
DiffStack is an autonomous driving stack with modules for prediction, planning, and control (Fig. 2). At a high level, DiffStack produces a trajectory for the AV follow given goal state, the tracked states of other agents, and a lane graph. To do so, it predicts where non-AV agents could move in the future; uses the predictions select safe, low-cost, and dyyamically--easiblle motion plan from set of candidates; and finally optimizes the plan to yield a continuous and smooth ego- trajectory.
The key feature of DiffStack is that it is differentiable, i.c., we can compute /0,, the gradient of the training objective Lj of downstream module j with respect the parameters O, of upstream module i. Differentiabillty enables training all parameters jointly for the overall objective of the stack with gradient- peadient-based optimizer. We also interpret DiffStack neural network with interpretable internal representations and fixed computational blocks for planning and control. Section 3. will describe how these components are made differentiable.
The main use considered in this paper is to train the prediction module with respect to a final control objective. Different agents' predictions have different importance to the ego-vehicle (sce examples in Appendix B). Standard metrics used to train prediction models are agnostic to the consequences of downstream planning, treating harmless and dangerous errors equally [26. 27]. By training directly for the end-objective of ego control, we expect DiffStack's 's prediction module to not only be accurate, but also synergize with planning and yield better overall ego controls.
We consider two open-loop training settings. reinforcement learning (RL) experiments we optimize a hindsight cost that measures the quality of an ego trajectory in hindsight, i.c., after observing the future trajectory of other agents in the scene. The hindsight cost can be viewed as a negative reward for RL. In imitation learning (IL) experiments we compare the planned ego trajectory to the ego-vehicle's enacted motion in the data through mean-squared-erroo (MSE) loss. We additionally explore using DiffStack to tune the hand-specified planning and control cost functions. In this section, we describe each module in DiffStack and their respective training setups, with more details in Appendix C.
Nomenclature. Ego refers to the AV and agent is non-AV vehicle pedestrian. States, consist of 2D position, heading, and longitudinal velocity. Control variables, u, are heading rate and longitudinal acceleration. A trajectory is sequence of states, or states and controls, depending on the context. Distance between states is measured by the 2D Euclidean distance, whereas root-mean- -sqaans state distance over time is used for trajectories. We denote trainable parameters of the prediction module by 0, and trainable parameters of the planning and control cost by w. We perform two types of optimization: training (also called learning) optimizes and w to minimize a loss L over some data; planning and control optimizes the ego trajectory to minimize cost function with 0 and w fixed.
3

---

3.1 DiffStack modules
Prediction. We employ Trajectron++ [42], state-of-the art CVAE that takes H seconds of state history for all agents as input, and outputs multimodal trajectory predictions for one agent a E A,
s(iTT) = (0))kK = CVAE - :))
(1)
where l K is the mode of the output distribution. We will use 8, = 5a (0) for brevity. The encoder of the CVAE processes agent state histories with recurrent LSTTM networks and models inter-agent interactions using graph-based attention. The decoder is GRU that outputs a Gaussian Mixture Model (GMM) for each future timestep. The GMM modes correspond the CVAB's K 25 discrete latent states. To ensure predictions are dyynamccally-fasiii and then integrated through known (differentiable) dynamics function to produce trajectory. We use the default model configuration without map and ego conditioning. We augment the input states with an ego-indicator variable to allow for ego-agent relation reasoning. The raw prediction training objective is the Info VAE loss, Ppeed = LInIoVAE ($a, sa). the same as for the original Trajectron++.
GMMs are defined over controls
Planning. The planner is a sampling- based algorithm that generates a set of N dynamically -ynamically-fasibble feasible ego trajectory candidates, P= (8n; Uh Jue N. and selects the candidate with the lowest cost. Namely,
Splan: Uplan arg min C(sn: Uny Sue 9, m. w), Sr
(2)
where C' is the cost function, Sag are multimodal predictions for all agents from the prediction module, g is given goal, and m lane graph. To generate trajectory candidates we sample a set of lane-centric terminal states, fit cubic spline from the current state to the terminal state, and reject ceaamically-iffessbbe trajectories. The cost function is weighted sum of handcrafted terms, C(:: w) = wi Cooll (s, SaEA) + w2Cgg(s g) + WyyCe (s, m) + Wi m) + ugCC(();
(3) penalizing collisions, distance to the goal, lateral lane deviation, lane heading deviation, and con- trol effort, respectively. Most notably, the collision term incorporates predictions into planning, Ceon(s, R = Ens E şo EREK Tx||s(()) - (12 where 8a,k is the k-th mode of the predicted trajectory distribution for agent a, T& is the probability of the k-th mode, &p is a Gaussian radial basis function, and ||- is the Euclidean norm. Without loss of generality, in experiments we only do prediction for the vehicle closest to ego, and use GT futures for other agents. The remaining terms of (3) are defined in Appendix C.
We make the planner differentiable by relaxing the arg min operator of (2) into sampling from a categorical distribution, and choosing cross entropy loss for Lplan. effectively treating the planner as a classifier during training. Specifically, we define the probability pn of choosing candidate n as = exp(- Uni //Z where 3 is temperature parameter and Z is the normalization constant. The planning loss is then Lplan LCB - -Enen 1s,, = s*) sog Pag where s* is target trajectory chosen depending on the training setup (see Section 3.2). We can now compute &Cylaa &Ce áp.. &C
ow Sp, 5C &u and similarly, 6Lylum = &Cer 5p., &C 5Ccall éa, where all terms exist.
8#
6p,. 8C 5Ceall &n g.
Control. The control module performs MPC over a finite horizon using an iterative box-constrained linear quadratic regulator (LQR) algorithm [43]. Formally, aim to solve
Metr arg min C(s, u; Aa A. 9, m; m.w) s.t. (0) -niti,, init, ((++1)== = fa( (1)),, uSusT, (4)
Setr:
where C denotes the cost function, fa the dynamics, iiit the current ego state, and u, u the control
limits. We use the cost defined in (3) for C' and the synamically-cxxendd
unicycle [44] for fa. We
initialize the trajectory with Uplan from the planner. The algorithm then iteratively forms and solves
quadratic LQR approximation of (4) around the current solution (()), ( for iteration i, using first- and second-order Taylor approximations of fa and C., respectively. The trajectory is updated to be close the LQR optimal control while also decreasing the original non-quadratic cost. We stop iterations upon convergence or a fixed limit.
To make the control algorithm differentiable we leverage Amos et al. [16]. The iLQR optimal trajectory, Sar. can be differentiated wrt. C and fa by implicitly differentiating the underly- ing KKT conditions of the last LQR approximation. The gradients can be analytically com- puted by one additional backward pass of modified iterative LQR solver. If iLQR fails to converge, we do not backpropagate gradients. In our setting fa is fixed. We compute gradi- ents wrt. cost parameters, SCaz
8Lat drmg &C
5aete &C dw and further wrt. prediction model parameters,
Am
&Cau 8Lms do.u SC &Co. S. 59
dear 6C 8C Φ. where Lete is the control loss, which we will discuss next.

---

Table 1: Open-loop evaluation results. DiftStack outperforms all baselines in planning and control metrics because it trains its prediction model for the downstream task.
Method
Pred. End Obj.
ADE (m)
NLL (nats) + Std. Err.
Planning Loss
Hindsight Cost J
Obj.
+ Std. Err.
x10 Std. Em.
x10-3. Std. Err.
No prediction
0.00 0.00
0.00 + 0.00
Standard
1.32 + 0.06
0.42 + 0.04
-3.76 0.23
-1.61 0.05
Distance weighted
1.34 0.06
1.19
-4.04
-1.68 = 0.04
/
VCost weighted [27]
1.43 0.10
-.50
-4.29
-1.77 0.07
DiftStack (no Greed)
/
1.37 = 0.06
8.01 0.75
-5.13
-1.87 0.04
DiffStack
/
1.27 = 0.07
0.38
-.18
-1.86
GT prediction
0.00
-2.25
3.2 End-to-end training
An important question for data-driven AV stacks the training objective and data. Learning in the real world is prohibitively expensive, and building a simulator with realistic traffic agent behavior an open challenge. Accordingly, standard practice is to perform open-loop training with human driving data [45, 13, 31, 23, 24, 46, 47]. We consider two common types of open-loop training settings: reinforcement learning (RI.) and imitation learning (II.).
In the RL setting, we aim minimize the (hindsight) cost of output ego trajectories. Setri Mctrs over dataset, Letr L Ch (Setr, Metr; 5tt A: 9: m. w) The hindsight cost C# captures the quality of a trajectory hindsight, i.c., after knowing the future trajectory of non-ego agents s Sae similar to the concept of rewards in RL. We choose Cи identical to the control cost C defined in (3), but with GT future trajectory inputs instead of predictions, and fixed w. Note that our open-loop training setup does not account for the effect of ego actions on other agents; nevertheless, we treat recorded trajectories as GT futures, as the prediction literature [48]. In this setting we only train the prediction model. Given GT futures oCue choose the trajectory candidate with the lowest hindsight cost, s* arg min, mim.,,,...e Cи (s Uni -).
ACHe [ where both terms exists. For the planner's target we 69
59 5(
In the IL setting we aim minimize the MSE between the output ego trajectory and the GT in the dataset, Letr = Lu. = Ex=1:r ||scer - 182. For the planner's target we choose 8* =8t
The total loss for training DiffStack is L= C=ai Epred + a2 Cp + azCetr. We experiment with different a values, including setting each a to zero. In the following we omit a, for brevity.
4 Experiments
We ask the following questions: 1) can DiffStack learn predictions that lead to better plans? 2) how does DiffStack compare to alternative planning-aware training techniques? 3) can DiffStack correct integration errors? 4) can DiffStack learn cost from imitation? 5) do our results translate closed-loop simulation? We begin with open-loop experiments (1-4), and then present closed-loop results (5) in Section 4.3.
4.1 Experimental setup
Dataset. We the nuScenes dataset [17], comprised of state annotations for vehicles and pedestri- ans in 1000 scenes across Boston and Singapore. For training and open-loop evaluation we sample suitable planning scenarios from the dataset with H = 4s history and 7' future data. For each scenario, we choose one vehicle to act as the ego. Details including dataset splits are Appendix D.
Metrics. We evaluate DiffStack's prediction module via standard prediction metrics: average displacement error (ADE) for the most-likely prediction and negative log- likellhoood (NLL) for its full distributional output. To evaluate planning and control, we use the cross-entropy planning loss Lplan = LCE, and control loss Letr- We report the hindsight cost for reinforcement learning (Cour CHC) and MSE for imitation learning experiments (Ceur (MSE). Since control an AV stack's 's end-goal, Letr reflects the overall performance of the stack. To make these metric values more interpretable, report them relative No prediction baseline and GT prediction- based oracle. The bascline ego plans without predictions (ignoring the predicted agent), providing a performance lower bound. The GT oracle plans with GT futures in place of predictions, providing notion of
5

---

fulure
ego, pian
predictions
drivaktle _ares
lane
lane_ divide
wallwayy
ped. prossinng
stop, Jine
roed, divider
roed segment
Standard
DiffStack
Standard
DiffStack
Figure 3: Left: The standard stack predicts an unlikely lane intrusion with high confidence, causing the planner to swerve. In contrast, DiffStack's ' prediction assigns most of the probability mass to reasonable trajectories that consider the presence of the ego vehicle, and the resulting motion plan stays well within lane boundaries. Right: The standard stack incorrectly predicts that vehicle would collide into the ego-vehicle from behind with high probability, causing the ego-vehicle to move into the intersection. DiffSStack assigns more probability to the agent slowing down, as result the ego-vehicle slows down before the intersection.
performance upper bound. All results are averaged over our validation set, and we report standard errors of the mean over 5 training seeds.
Baselines. We compare DiffStack with standard AV stacks. For a fair comparison, we use the same set of modules as in DiffStack, but without making the planner and controller differentiable. We only train the prediction model, using increasingly planning-aware training objectives. Standard trains the prediction model with only the prediction loss pred, unaware of downstream planning. Next, we re-weight prediction losses (in each batch) based on a handcrafted measure of relevance for planning. Distance weighted weights losses with the inverse distance between the ego and agent GT futures. Cosst weighted uses the magnitude of the control cost gradient with respect to the GT future trajectory of the agent, proposed as a planning- aware prediction metric in [27]. Finally, DiftStack backpropagates gradients of the final loss to the prediction module. DiffStack (no ppred) uses L= Eplan + Letr; DiffStack uses C= Ppeed + Lpann +
Letr-
Implementation details. We implement DiffStack in PyTorch [49] and build on the open-source code of Trajectron++ [42] and Differentiable MPC [16]. We use H 4s, T= =3s, and 0.5s. We train all models for 20 epochs using NVIDIA Tesla V100 GPUs, taking 10-20 hours. Additional details are in Appendix C. The code is available at https: http:/ssitsssooooeeeee www/viee/diffftteeeeeeeeeeeeeeee
4.2 Open-loop results
End-to-end training is useful. Our main results are summarized in Table 1. The key observation is that DiffStack improves the effectiveness of predictions on ego planning by up to 15.5% compared to standard training (Ceur = - 1.86 vs. - -1.61); and reduces the gap to the cost attainable with a GT-informed oracle by 39. 19.1% 1% 1.86 + 2.25 vs. -1.68 2.25). DiffStack also improves planning without significantly impacting raw prediction accuracy (ADE=1.27 vs. 1.32, within standard error). Alternative methods that re-weight the prediction loss in planning-awe
manner (rows 2 and 3) also help, but less than DifiStack. Surprisingly, can even recover comparable prediction performance in terms of ADE when training solely for planning and control objectives (row 4). The poor distribution fit (high NLL) is due to control cost being agnostic to the variance of the predicted GMMs.
DiffStack makes fewer prediction errors that affect planning. Qualitatively analysing predictions and plans shows that the main source of improvement from end-to-end training is the reduction of spurious and/or unrealistic predictions that lead to plans with large (unnecessary) deviations from the lane center or the goal position. Fig. 3 shows two particular examples. In both cases, DiffStack's predictions are more realistic and accurate, yielding much more reasonable downstream plans.
DiffStack can compensate for system integration errors. Integrating independently developed modules into an AV stack can challenging due to, e.g. misaligned module interfaces. Table shows results for an experiment that explores this issue. We introduce an artificial interface mismatch between prediction and planning by adding a fixed Im offset all GT prediction targets. As a result, the standard model yields very poor plans (note the positive value in row 1, indicating worse performance than baseline planning). However, DiffStack can compensate for interface mismatch and substantially improves the overall plans, suggesting that it can learn to correct erroncous predictions that affect planning. More broadly, these results demonstrate the potential for differentiable AV stacks

---

Table 2: System integration results. DiftStack compensates for errors in module integration. Even if GT prediction targets are erroneously offset by Im, training with downstream losses corrects the prediction model's 's bias and improves overall system performance.
Method
Pred.
End
ADE (m) 4
NIL (nats)
Planning Loss 4
Hindkight Cost
Obj.
Obj.
- Std. Err.
+ Std. Err.
x10 + Su.. Em.
x10 _ Std. Emr
No prediction
0.00 1 0.00
0.00 1: 0.00
Slandard-biased DiffStack-biased
V biased
3.64 + 0.34
8,43 1 0.72
16.22 - 2.73
3.13 = 0.40
v biased
2.01 + 0.08
4.20 + 0.09
-3.58 + 0.96
-1.61 + 0.12
GT prediction
-7.56 + 0.00
-2.25 0.00
Table 3: Imitation learning results. Diff'Stac'ss plans better match the GT ego trajectories from data.
Method
Pred.
End
ADB (m) 4 + Sud. Err.
NLL (nats) 4 + St.. Er.
Planning Loss 4
MSF (m²) +
Obj.
Obj.
x x10- + Std. Er.
x10-2, Std. Err.
No prediction
0.00 + 0.00
0.00 +: 0.00
Standard
1.25 = 0.02
0.53 + 0.23
-0.50 0.05
0.60 +: 0.15
DiffStack
1.43 + 0.14
0.30 + 0.23
-0.80 t 0.04
-1.80 + 0.16
GT prediction
11.10 0.00
-2.00 + 0.00
to reduce various development/engineette costs associated with developing AVs, e.g., by replacing tedious manual parameter tuning with end-to-end data-driven optimization. We further explore this topic in the cost tuning experiments below.
DiffStack can imitate humans better. Imitation learning results are summarized in Table 3. The overall performance of the stack is now measured by Letr = LMSE. the MSE between planned and human expert trajectories relative to the MSE for the baseline planner (without predictions). We observe a similar trend as before; however, the results also reveal some limitations. The standard stack performs worse in terms of MSE than the baseline stack which neglects predictions entirely (note the positive sign for relative LMSE in row 1). While DiffStack improves MSE significantly, the absolute difference is small. We hypothesize this is caused by our cost function not being rich enough to fully capture human driving behavior; and because agent-agent interactions that affect planning rare in nominal driving data (comprised mainly of lane- and speed- -reeedk.eeing...... These limitations could be addressed by richer cost function and alternative goal definitions, which we leave to future work.
DiffStack can learn better control cost. While our main use case in this paper is to learn planning- aware prediction, DiffStack opens up various other opportunities for data-driven optimization of various components of the AV stack. For example, an impor- tant challenge in practice is design cost function for planning and control. In this experiment, we explore DiffStack's ' potential to learn interpretable control costs that optimize the quality of output plans from data. Table 4 reports results in the imitation learning setting, where we first train a prediction module (as before), then we train for an additional 20 epochs allowing DiffStack to update the weights w; of the control cost (3) by backpropagating gradients from the final control loss Letr- Compared DiffStack significantly decreases plan MSEs (row 2). The resulting learned weights w. are lower for the control effort term, and higher for the goal and lane keeping terms (see Appendix A). To ensure safety, we fix the weights for collision avoidance. We leave comparison with alternative techniques for learning control costs to future work. We expect DiffStack to be more effective compared to, e.g.. Bayesian optimization, where the number of learned parameters is large.
Table 4: DiffStack learns a better control cost.
Method
Plamning Loss 4
MSE (m²)
Hand-tuned
2.67 + 0.00
0.38 + 0.00
Leamndd
2.41 + 0.01
0.32 + 0.00
the default hand-tuned weights (row 1),
Ablations. Results of an ablation study can be found in Appendix A. In short, improvements from DiffStack are consistent with our observations when training for L = Lpeed + Lplan L- Cpeed + Leur when changing the relative scale of loss components; with higher time resolution At= 0.1 for planning and control; and when only using one of the planning or control modules.
4.3 Closed-loop evaluation
Simulation. We perform closed-loop simulation using simple log-replay setup, where ego states are unrolled based on the planned control outputs and known dynamics, while non-ego agents follow their fixed trajectories recorded in the dataset. The evaluation scenarios are Tam 10s long. The goal
7

---

Table 5: Closed-loop results. DiftStack outperforms the standard stack similarly open-loop results.
Method
Trajectory Cost Open-loop Cost
Collision Cost Lane Keep Cost
Control Effort
Deviation (m)
x x10
x10-2
x10 4
x10 +
x x10
x x10
No prediction
0.00
0.00
0.00
0.00
0.00
0.00
Standard
0.87
0.33
-8.21
8.91
0.18
1.05
DiffStack
4.37
-2.4
-.20
4.84
-0.99
0.98
GT prediction -6.30
-258
-8.54
3.34
-1.08
0.70
and lane inputs for planning updated in each simulation step based on the logged ego trajectory. We compute the following metrics. Trajectory Cost captures closed-loop performance: it evaluates
the cost function C' on unrolled simulation trajectories, 1/T Σi=1-Tm C(ssun Pimm V
(t) & S... m!!))
where Ssim and Ugim are the unrolled ego state and control. We exclude the goal cost term because the goal is updated in each simulation step. Open-loop Cost is the average of hindsight costs calculated open-loop at each simulation step, Luc
CC( = Metr; A 9, m). Collision Cost is the s
collision term in the trajectory cost, w² Ccon ( sim SaEA) Lane Cost is the sum of lane keeping terms, Wyy Caa W4CC Control Effort is the control effort term, W7s Cu. Deviation is the average
gt,(())
distance between unrolled and logged ego trajectories from the data, []s4)) -..
Results. Simulation results are in Table 5. Most importantly, we observe similar relative improvement for DiffStack in terms of closed loop trajectory cost in terms of open-loop cost. DiffStack performs substantially better that the no prediction baseline, but somewhat surprisingly the standard stack performs worse, both in terms of closed-loop trajectory cost and open-loop cost. Analysing the cost components sheds light on possible reasons. As expected, incorporating predictions into planning results in lower collision costs, but higher lane keeping costs. The contribution of these two terms to the average cost depends on the data distribution, e.g., the frequency of close interactions where predictions are useful. As we saw earlier in Fig. 3, poor predictions in the standard stack frequently lead to unnecessary lane deviations in the planner. This is reflected in the substantially higher lanc cost and control effort cost for the standard stack compared to DiffStack in Table 5.
5 Limitations & Conclusions
Limitations. One limitation of our work is the open-loop training and log-replay based simulation setup. In lieu real-world AV simulator with strong behavioral realism, this is standard practice; however, recent efforts on accurate behavior simulation could be leveraged in the future [50, 51, 52, 53, 54]. Our implementation of DiffStack also has limitations. First, it is not differentiable wrt. all possible parameters, e.g., no gradients flow from the control loss to the planner's 's trajectory candidate generator. Future work may develop more sophisticated differentiable algorithms and explore ideas for gradient approximation for non-differentiable components [55]. Second, individual modules could be improved, e.g., by adding a more sophisticated prediction model, improving candidate sampling in the planner, and adding trust-region constraints the controller. Finally, even though hand-engineered components, modularity, and intermediate training objectives in differentiable stacks remedy challenges of learning AV policies end-to-end, other challenges naturally remain. For instance, designing an overall performance metric, or reducing the scarcity of and cost to obtain (interesting) driving data remain open problems, each of which are impactful areas of future work.
Conclusions. In this paper we take a step towards fully-lifferentiablee and modular AV stacks by introducing key differentiable components for prediction, planning, and control. Our experimental results show the potential benefits of jointly training modules of AV stacks for downstream per- formance. For motion prediction in particular, our results indicate that there is value in moving from purely prediction-orient with arguments in recent work [26. 27, 28]. While our experiments focused on learning planning- aware predictions, DiffStack opens up various exciting opportunities for task-oriented learning in modular stacks. For example, we may learn rich neural network as part of the control cost to capture -o-eineer concepts, e.g., reasoning with occlusions or accounting for uncertainties. Eventually, we envision having differentiable modules for the entire AV stack, allowing any subset of modules be learned and optimized for downstream task. Overall, this would relax information bottlenecks and enable uncertainty to more casily propagate through the stack without needing to forgo interpretability, modularity, and verifiabilty of the various components.
evaluation metrics towards downstream task-oriented metrics, in line
8

---

Acknowledgments
We thank Yuxiao Chen and Edward Schmerling for generous help with our planner implementation and closed-loop evaluation experiment; Christopher Maes for fruitful discussions differentiable MPC and cost learning experiments; and Nikolai Smolyanskiy for valuable high-level feedback.
References
[1] General Motors, Self-driving safety report, 2018. Available at https://sww. gm wtw.gg...ooooootttteeeeeeee company/docs/ust /ot///////////////////e///e/eeee en/ gmcom/ (gmsafetyrpporrrr pdf.
[2] Uber Advanced Technologies Group. A principled approach to safety, 2020. //uber. app. http:/..o////bbeerrrttaaaat//eee
Available at https:
[3] Lyft. Self-driving safety report, 2020. Available at http:://2ggllazoooorgggggo2222 upeeeggne...........mee netdna- ssl. com/wp-
pdf.
[4] Waymo. Safety report, 2021. Available httpss///arrro.cco/////ttttatasp Retrieved July 4, 2021.
Argo AI. Developing self-driving system system you trust, Apr. 2021. Available https://www.
Motional. Voluntary safety self-assessmennt, 2021. Available https://drive.. wtwp:/rrriveeogg...o.o
[7] Zoox. Safety report volume 2.0, 2021. Available htttps://2ooxx.cmm/
Self-driving safety // ('images..ny .wiiii. wwwm//ooteettseef--rrrrr. http://eelf-dddiin--eeeee watee/aaaaa/eaaeeeeeeeeaeeeeeeee auto iito-print-as woolf-driving safety-repors 2021 2021-update update pdf.
NVIDIA.
report, 2021.
Available
https:
[9] X. Zhou, V. Koltun, and P. Krähhenüühl. Tracking objects as points. In European Conference on Computer Vision, 2020.
[10] X. Weng, B. Ivanovic, and M. Pavone. MTP: Multi-hypothesis tracking and prediction for reduced error propagation. In JEEE Intelligent Vehicles Symposium, 2022.
[11] X. Weng, B. Ivanovic, K. Kitani, and M. Pavone. Whose track is it anyway? Improving robustness to tracking errors with affinity-based trajectory prediction. In Conference on Computer Vision and Pattern Recognition, 2022.
[12] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving arXiv preprint 1604v:1604.07.. 2016.
[13] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In Conference on Computer Vision and Pattern Recognition, pages 8660-8669, 2019.
[14] S. Casas, Sadat, and R. Urtasun. MP3: A unified model map, perceive, predict and plan. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] P. Karkus, X. Ma, D. Hsu, L. P. Kaclbling, W. Lec, and T. Lozano-Pérez. Differentiable algorithm networks for composable robot learning. In Roboticx: Science and Systems (RSS), 2019.
[16] B. Amos, Jimenez, Sacks, Boots, and Kolter. Differentiable MPC for end-to-end planning and control. In Advances in Neural Information Processing Systems, pages 8299-8310, 2018
[17] H. Caesar, Bankiti, A. H. Lang, Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuScenes: multimodal datasct for autonomous driving. In Conference on Computer Vision and Pattern Recognition, 2020.
[18] B. Ivanovic, K.-H. Lee, P. Tokmakov, B. Wulfe, R. McAllister, A. Gaidon, and M. Pavone. Heterogencous- agent trajectory forecasting incorporating class uncertainty. arXiv preprint 1rX1v:1104.12466, 2021. URL https: //t///axxvvv org/ abss//14 12446.
[19] X. Weng, Y. Yuan, and K. Kitani, PTP: Parallelized tracking and prediction with graph neural networks and diversity sampling. IEEE Robotics and Automation Letters, 6(3):4640 4644, 2021.

---

[20] B. Ivanovic, Y. Lin. S. Shrivastava, P. Chakravarly, and M. Pavone. Propagating state uncertainty through trajectory forecasting. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[21] J. Ziegler, P. Bender, T. Dang. and C. Stiller. Trajectory planning for BERTHA A local, continuous method. In IEEF Intelligent Vehicles Symposium, 2014.
[22] C. Liu, S. Lee, S. Varnhagen, and H. E. Tseng. Path planning for autonomous vehicles using model predictive control. In IEEE Intelligent Vehicles Symposium, 2017.
[23] B. Ivanovic, A. Elhafsi, G. Rosman, A. Gaidon, and M. Pavone. MATS: An interpretable trajectory forecasting representation for planning and control. In Conference on Robot Learning, 2020.
[24] Y. Chen, U. Rosolia, C. Fan, A. D. Ames, and R. Murray. Reactive motion planning with probabilistic safety guarantees. In Conference on Robot Learning, 2020.
[25] S. Schaefer, K. Leung, B. Ivanovic, and M. Pavone. Leveraging neural network gradients within trajectory optimization for proactive human-robot interactions. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9673 9679, 2021.
[26] S. Casas, C. Gulino, S. Suo, and R. Urtasun. The importance of prior knowledge in precise multimodal prediction. In International Conference on Intelligent Robots and Systems (IROS), 2020.
[27] B. Ivanovic and M. Pavone. Injecting planning- IEEE Intelligent Vehicles Symposium, 2022.
planning-awareness awareness into prediction and detection evaluation. In
[28] R. McAllister, B. Wulfe, J. Mercat, L. Ellis, S. Levine, and A. Gaidon. Control-awart prediction objectives for autonomous driving. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[29] A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani. Deep reinforcement learning framework for au- tonomous driving. Electromic Imaging, 2017(19):0-76,, 2017.
[30] A. Jain, L. Del Pero, H. Grimmett, and P. Ondruska, Autonomy 2.0: Why is self-driving always 5 years away? arXiv preprint arXiv:2107.08144 2021.
[31] Zeng, S. Wang, R. Liao, Chen, B. Yang. and Urtasun. Dsdnet: Deep structured self-driving network. In European Conference on Computer Vision, pages 156-172 Springer. 2020.
[32] A. Cui, S. Casas, A. Sadat, R. Liao, and R. Urtasun. Lookout: Diverse multi-future prediction and planning for self-driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16107-16116, 2021.
[33] R. Jonschkowski, D. Rastogi, and O. Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. Proceedings of Robotics: Science and Systems, 2018
[34] P. Karkus, Hsu, and W. S. Lee, Particle filter networks with application to visual localization. In Proceedings of the Conference on Robot Learning, pages 169-178, 2018.
[35] K. M. Jatavallabhula, G. Iyer, and L. Paull. gradSLAM: Dense SLAM meets automatic differentiation. In International Conference on Robotics and Automation (ICRA), pages 2130-2137, 2020.
[36] P. Karkus, S. Cai, and D. Hsu. Differentiable SLAAM-ntt: Learning particle SL.AM for visual navigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[37] P. Karkus, D. Hsu, and W. S. Lee. QMDP-net: Deep learning for planning under partial observability. In Advances in Neural Information Processing Systems, pages 4697 4707, 2017.
[38] S. Gupta, J. Davidson. S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. arXiv preprint arXiv: 1022.0200,. 2017.
[39] M. Okada, L. Rigazio, and T. Aoshima. Path integral networks: End-to-end differentiable optimal control. arXiv preprint arXiv: / 0000000000000 09590,099777111 2017.
[40] M. Pereira, D. D. Fan, G. N. An, and E. Thcodorou. MPC-inspired neural network policies for sequential decision making, arXiv preprint / 002200503, 2018.
[41] S. East, M. Gallieri, J. Masci, J. Koutnik, and M. Cannon. Infinite-horizon differentiable model predictive control. In International Conference Learning Representations, 2019.
[42] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-fesibbe trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, 2020.
10

---

[43] Y. Tassa, N. Mansard, and E. Todorov. Control-limiter differential dynamic programming. In IEEE International Conference Robotics and Automation (ICRA), pages 1168- 1175, LEEE, 2014.
[44] S. M. LaValle. Better unicycle models. In Planning Algorithms, pages 743-743. Cambridge University Press, 2006.
[45] M. Bansal, A. Krizhevsky, and A. Ogale. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst. In Robotics: Science and Systems, 2019.
[46] Y. Chen, Rosolia, Ubellacker, Csomay- Shanklin, and D. Ames. Interactive multi-modal motion planning with branch model predictive control. Robotics and Automation Letters, 7(2):5365-5372, 2022.
[47] Y. Chen, B. Ivanovic, and M. Pavone. ScePT: Scene-consistent, policy-based trajectory predictions for planning. In Conference on Computer Vision and Pattern Recognition, 2022.
[48] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras. Human motion trajectory prediction: A survey. International Journal of Robotics Research, 39(8):895 9358895-933,, 2020.
[49] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury. G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
[50] L. Bergamini, Y. Ye, O. Scheel, L. Chen, C. Hu, L. Del Pero, B. Osiáski, H. Grimmett, and P. Ondruska. SimNet: Learning reactive self-driving simulations from world ohservations. In IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2021.
[51] S. Suo, S. Regalado, S. Casas, and R. Urtasun. TrafficSim: Learning to simulate realistic multi-agent behaviors. In Conference on Computer Vision and Puttern Recognition (CVPR), pages 10400-10409, 2021.
[52] Н. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. M. Wolff, A. H. Lang, L. Fletcher, O. Beijbom, and S. Omari. NuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles. In Conference Computer Vision and Puttern Recognition (CVPR) ADP3 Workshop, 2021.
[53] D. Rempe, J. Philion, L. J. Guibas, Fidler, and O. Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[54] D. Xu, Chen, B. Ivanovic, and M. Pavone, Bits: Bi-level imitation for traffic simulation, arXiv preprint arXiv:2208. 12403, 2022.
[55] T. Weber, N. Heess, L. Buesing, D. Silver. Credit assignment techniques stochastic computation graphs. arXiv preprint 10X1v:1901.0110 2019.


Reviews:
[{"review_id": "vpIUi2eKhlrh", "reviewer": "Reviewer_a5FX", "summary": "The authors tackle the task of offering a full autonomous driving software stack. In contrast to classical approaches that focus on modular software containing of perception, planning and control, the authors present here software components that are full differentiable. The authors focus on prediction, planning and control modules that are made differentiable, train it in a simulation environment (nuScenes) and then show the results in comparison to baselines. The authors display that this approach - in contrast to prior End-to-End methods - gives more modularity and interpretability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-The paper is well-written and good to read, the technical explanations are exceptional\n- The paper tackles a very hot topic in the field, especially for AVs were we are currently seeing bottlenecks in hand-engineered algorithms\n- Honest opinion about the limitations of this approach, it is not a full full differentiable stack\n- The results indicate that \"there is value in moving from purely prediction-oriented evaluation metrics towards downstream task-oriented metrics\" -> This was new to me and is a significant result, especially with all the work in prediction-algorithms right now\n\nWeaknesses:\n- No hardware experiments were provided\n- No Training results were provided. Although it is stated that the nuScens datasets is used there is nothing explained about training quality and training time\n- No comparison to a modular software stack (classical pipeline) or a complete end-to-end pipeline. I do not understand the comparison with the \"baseline\" (We compare DiffStack to the same prediction-planning-control stack, but with modules\n219 trained for different objectives that capture a varying degree of planning-awareness) ??\n- The authors claim with the title its a full-stack but in the limitations, they show that this approach is only performed in open-loop training and evaluation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "For me this paper is a strong accept since the paper contains many components and ideas that will potentially have a major impact in the field of autonomous vehicles. The paper is both well-written and technically deep, the authors explain whats going on in their different Software Stack Modules (Predction, Planning, Control) and how they are training in and end-to-end fashion. Generally that paper contains all important components and is of high interest for the community", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "The biggest issue i have with the paper are the results and explanations of the outcome and why diffstack is better. Since the baseline is also diffstack but trained with different objectives (correct?) I do not see the justification for the discussion. The discussion section is great and shows why Diffstack is good, but I miss the comparison to a classical stack or a real end-to-end pipeline"}}, {"review_id": "Yos-FgeLhk", "reviewer": "Reviewer_jUVa", "summary": "Rather than building an AV stack in a modular fashion, this work introduces a differentiable and modular stack for prediction, planning and control. This enables optimization of upstream components such as prediction via backprop through planning and control.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is still interpretable in the sense that it is composed of modules with specific purposes. \n- The method is able to be trained end-to-end due to being fully differentiable. \n- Comparison to a non-differentiable but modular method is made clearly.\n\nWeakness:\n- no real world or even simulator results.\n- not differentiable to all possible parameters\n- Comparison to an end-to-end neural network is not made. The authors note that not all parameters are differentiable. How does this system compare to an end-to-end neural network that doesn't attempt to provide an interpretability? How much performance is being sacrificed for interpretability? Can the authors provide quantitative or qualitative examples of how this interpretability is useful?\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Paper is well written and well motivated. \nThe experimental section could be made much strong through additional baselines (particulary a non-modularized end-to-end system), and real world or even simulated experiments.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- The argument for a fully data-driven yet modular and interpretable AV architecture is compelling in general, but could be better articulated in this work. Particularly there is no comparison or demonstration of their approach providing a benefit  over a non-interpretable end-to-end system.  It would be great if the authors could find some way to quantify this, or at least qualitatively demonstrate the difference. \n\n"}}, {"review_id": "GUmbizgIYmj", "reviewer": "Reviewer_67Mn", "summary": "The authors present a fully differentiable yet modular pipeline for autonomous vehicle perception and control. It preserves the traditional abstractions such as perception, prediction, planning, and control but implements differentiable versions of these components so that gradients can pass all the way from task performance back to perception. They demonstrate the value of this on a trajectory prediction task. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n - The idea is well-motivated and the approach seems well executed. The approach preserves some of the important interpretability charateristics of traditional methods while still enabling end-to-end learning. \n - Using task performance to learn perceptual models is a promising area of research, and this paper showcases a possible method for doing so in the case of autonomous vehicles. \n\n\nWeaknesses / Questions:\n\n - The authors should highlight to what degree the differentiable components are novel. To my understanding they have mostly taken existing components and assemble them together so the contribution should be mostly considered at the system level. If this is not the case the authors should clarify. \n\n - The cost function in (3) contains a lot of weights. How sensitive is the algorithm to the selection of these weights? Similar question for $\\alpha_{1:3}$ in the combined loss function.\n\n - How is the discretization of $s$ chosen for the planner (i.e., what is the dimension of the Categorical distribution and why)?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the work presents a novel autonomous vehicle pipeline that is modular yet differentiable. Although none of the individual components seems to be particularly novel, the system that is constructed is and the advantages that it affords seem to be convincing.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "5TApbSwATqq", "reviewer": "Reviewer_oNtX", "summary": "This paper presents an end-to-end differentiable stack for autonomous driving. The prediction module is a neural network, and the planning and control modules are hand-designed algorithms. Notably, the hand-designed algorithms are differentiable, which allows training of the upstream prediction module for a downstream control objective by backpropagating gradients through these hand-designed algorithms. The authors make planning differentiable by replacing the argmin operation with sampling from a categorical distribution, and the authors make use of an off-the-shelf differentiable MPC algorithm to make control differentiable. The authors show open-loop, offline results which indicate that this kind of planning-aware training of the prediction module improves performance. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- A step towards end-to-end training of prediction modules, while retaining the interpretability / stability of hand-designed control and planning algorithms. \n- The paper is well-written and easy to parse. \n\nWeaknesses\n- Only offline / open-loop evaluation, so it is difficult to predict what real world performance will be like. \n- I found the metrics to be somewhat hard to interpret, but this could be because I’m not a researcher in the AV space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I do not conduct research in autonomous driving, so take my recommendation with a grain of salt (this is the main reason I selected weak accept instead of strong accept). Overall, I felt the paper was well-motivated and executed, and tackles what I believe is an important problem. I recommend acceptance. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "N/A", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)