PlanT: Explainable Planning Transformers via Object-Level Representations
Authors: Katrin Renz
Kashyap Chitta
Otniel-Bogdan Mercea
A. Sophia Koepke
Zeynep Akata
Andreas Geiger
Venue: CoRL 2022 Poster (2022)
Keywords: Autonomous Driving
Transformers
Explainability

Abstract: Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations containing all vehicle and road context information. In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. PlanT is based on imitation learning with a compact object-level input representation. On the Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3× faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf perception module provides a sensor-based driving system that is more than 10 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.

PlanT: Explainable Planning Transformers via Object-Level Representations
Katrin Renz 1.2
Kashyap Chitta 1.2 Otniel-Bogdan Mercea!
A. Sophia Koepke'
Zeynep Akata 1.3.3 Andreas Geiger
University of Tübingen Max Planck Institute for Informatics, Saarbrücken
Max Planck Institute for Intelligent Systems, Tübingen
https: //www. katr
katrinrrenn inninnienniiii detpaant
Abstract: Planning an optimal route in complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, sarning-based planners typically extract features from dense, nigh-dimensional grid representations con- taining all vehicle and road context information. In this paper, propose PlanT,, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. Plan'T is based on imitation learning with a compact object-level input representation. On the Longestó benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3 faster than equivalent bixel-based planning baselines during inference. Combining PlanT with an off thel- perception module provides sensor- based driving system that is more than 10 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision- making. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.
Keywords: Autonomous Driving, Transformers, Explainability
Introduction
The ability plan is important aspect of human intelligence, allowing to solve complex igation tasks. For example, to change lanes on a busy highway, a driver must wait for sufficient the expected behavior of the other vehicles. Hu-
space in the new lane and adjust the speed based mans quickly learn this and can generalize to new scenarios, a trait we would also like autonomous agents to have. Due to the difficulty of the planning task, the field of autonomous driving is shifting away from traditional rule-based algorithms [1, 2, 3, 4, 5, 6, 7, 8] towards learning-ban -baeed solu- tions [9, 10, 11, 12, 13, 14]. Learning- based planners directly map the environmental state represen- tation (e.g., HD maps and object bounding boxes) to waypoints or vehicle controls. They emerged as scalable alternative to rule-based planners which require significant manual effort to design.
Interestingly, while humans reason about the world in terms of objects [15, 16, 17), most existing learned planners [9, 12, 18] choose high-dimensional pixel-level input representation by rendering bird's eye view (BEV) images of detailed HD maps (Fig. left). is widely believed that this kind of accurate scene understanding is key for robust self-driving vehicles. leading significant interest in recovering pixel-level BEV information from sensor inputs [19, 20, 21, 22, 23, 24). In this paper, we investigate whether such detailed representations are actually necessary achieve convincing planning performance. We propose Plan'T, a learning-basse planner that leverages an object-level representation (Fig. right) as an input to a transformer encoder [25]. We represent scene as a set of features corresponding to (1) nearby vehicles and (2) the route the planner must follow. We show that despite the low feature dimensionality, our model achieves state-of-the results. We then propose a novel evaluation scheme and metric to analyze explainability which is generally applicable to any learning-bass based planner. Specifically, we test the ability of planner to identify the objects that are the most relevant to account for to plan a collision-free route.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Pixel-Lavel Representation
Scene
Object-Level Representation
Vehicle
ym12 wm1.2 2.1 yaw-1.3 spd=8
Vehicle
X=-9 y=11122112222 w=1 I= 1.2 yawm3 spd=
Route
Vehiclo
x-0 ymu7 w- h-2.2322233 you 2pdd
Ego
Route
0-5 y=1 we1lll112h2222-- yam-- id=0
Route
x= y- w= id=1
Figure 1: Scene Representations for Planning. As an alternative to the dominant paradigm of pixel-level planners (left), we show the effectiveness of compact object-level representations (right).
We perform a detailed empirical analysis of learning-based planning the Longestó bench- mark [26] of the CARLA simulator [27]. We first identify the key missing elements in the design of existing learned planners such as their incomplete field of view and sub-optimal dataset and model sizes. We then show the advantages of our proposed transformer architecture, including improve- ments in performance and significantly faster inference times. Finally, show that the attention weights of the transformer, which are readily accessible, can be used to represent object relevance. Our qualitative and quantitative results explainability confirm that PlanT attends to the objects that match our intuition for the relevance of objects for safe driving.
Contributions. (1) Using simple object-level representation, we significantly improve upon the previous state the art for planning on CARLA via PlanT, our novel transformer-based approach. (2) Through comprehensive experimental study, we identify that the ego vehicle's route, a full 360 field of view, and information about vehicle speeds are critical elements of a planner's input representation. (3) We propose a protocol and metric for evaluating a planner's prioritization of obstacles a scene and show that PlanT is more explainable than CNN-based methods, i.e., the attention weights of the transformer identify the most relevant objects more reliably.
2 Related Work
Intermediate Representations for Driving. Early work on decoupling end-to-end driving into two stages prediets a set of low-dimensional affordances from sensor inputs with CNNs which are then input to a rule-based planner [28]. These affordances are scene-descriptive attributes (e.g. emergency brake, red light, center-line distance, angle) that are compact, yet comprehensive enough to enable simple driving tasks, such as urban driving on the initial version of CARLA [27]. Unfortunately. methods based on affordances perform poorly on subsequent benchmarks in CARLA which involve higher task complexity [29]. Most state-of-the-ar driving models instead rely heavily on annotated 2D data either as intermediate representations or auxiliary training objectives [26, 30]. Several subsequent studies show that using semantic segmentation as an intermediate representation helps for navigational tasks [31, 32, 33, 34]. More recently, there has been rapid growth in interest in using BEV semantic segmentation maps as the input representation to planners [9. 12, 30, 18). To reduce the immense labeling cost of such segmentation methods, Behl al. [35] propose visual abstractions, which are label-efficient alternatives to dense 2D semantic segmentation maps. They show that reduced class counts and the use of bounding boxes instead of pixel- pixel-accurate masks for certain classes are sufficient. Wang et al. [36] explore the use of object-centric representations for planning by explicitly extracting objects and rendering them into a BEV input for a planner. However, so far, the literature lacks systematic analysis of whether object-centric representations are better or worse than BEV context techniques for planning in dense traffic, which we address in this work. We keep our representation simple and compact by directly considering the set of objects as inputs to our models. In addition to baselines using CÑNs to process the object-centric representation, show that using transformer leads to improved performance, efficiency, and explainability.
Transformers for Forecasting. Transformers obtain impressive results in several research ar- eas [25, 37, 38, 39], including simple interactive environments such as Atari games [40, 41, 42, 43, 44]. While the end objective differs, one application domain that involves similar challenges

---

to planning is motion forecasting. Most existing motion forecasting methods use a rasterized input in combination with a CNN-based network architecture [45, 46, 47, 48, 49, 50]. Gao al. [51] show the advantages of object-level representations for motion forecasting via Graph Neural Net- works (GNN). Several follow-ups to this work use object-level representations in combination with Transformer-basse architectures [52, 53, 54]. Our key distinctions when compared to these methods are the architectural simplicity of PlanT (our use of simple self-attention transformer blocks and the proposed route representation) well as our closed-loop evaluation protocol (we evaluate the driving performance in simulation and report online driving metrics).
Explainability. Explaining the decisions of neural networks is rapidly evolving research field [55, 56, 57, 58, 59, 60, 61]. In the context of self-driving cars, existing work uses text [62] or heatmaps [63] to explain decisions. In our work, we can directly obtain post hoc explanations for decisions of our learning-basee -baii Plan'T architecture by considering its learned attention. While the concurrent work CAPO [64] uses a similar strategy, it only considers pedestrian-ego interactions on an empty route, while we consider the full planning task in an urban environment with dense traffic. Furthermore, we introduce simple metric to measure the quality of explanations for planner.
3 Planning Transformers
In this section, we provide details about our task setup, novel scene representation, simple but ef- fective architecture, and training strategy resulting in state-of-the-ar performance. A PyTorch-style pseudo-code snippet outlining PlanT and its training is provided in the supplementary material.
Task. We consider the task of point-to- point-to-point navigation in an urban setting where the goal is to drive from start to a goal location while reacting to other dynamic agents and following traffic rules. We use Imitation Learning (II.) to train the driving agent. The goal of is to learn a policy æ that imitates the behavior of an expert x² (the expert implementation is described in Section 4). In our setup, the policy is a mapping π X' - W from our novel object-level input representation X' to the future trajectory W of an expert driver. For following traffic rules, we assume access to the state of the next traffic light relevant to the ego vehicle E {green, red].
Tokenization. To encode the task-specific information required from the scene, we represent it using a set of objects, with vehicles and segments of the route each being assigned an oriented bounding box in BEV space (Fig. right). Let & V.US,, where V. E IR x.x and S, E S represent the set of vehicles and the set of route segments at time-step & with A 6 attributes each. Specifically, if Oi, E X₁ represents a particular object, the attributes of os,t include an object type-specific attribute Zi. (described below), the position of the bounding box (Ii,e, I),,)) relative to the ego vehicle, the orientation Pi,t E [0, 2x), and the extent (wi,e, h.,r). Thus, each object oi, can be described as a vector o., = {Zz,e, Fs.e: Vi,ss Pi,t, Wi,e: }. or concisely as {o,,r.a)==
For the vehicles V/, extract the attributes directly from the simulator in our main experiments and use an off-the-shelf perception module based on CenterNet [65] (described in the supplementary material) for experiments involving full driving system. We consider only vehicles up distance D Dmaz from the ego vehicle, and use 01,,,1 (i.e., 2i,e) to represent the speed.
To obtain the route segments S,, we first sample a dense set of N₂ points U3 E N.2 along the route ahead of the ego vehicle at time-step t. We directly use the ground-truth points from CARLA as Ue our main experiments and predict them with perception module for the PlanT with per- ception experiments in Section 4.2. The points are subsampled using the algorithm [66, 67] to select subset út One segment spans the area between two points subsam- pled from the route, u, U1+1, E úde Specifically, o,,1,1 (i.e., 21,)) denotes the ordering for the current time-step t, starting from 0 for the segment closest to the ego vehicle. We set the segment length O,,,,6 ||,,t u.+1,ell2. and the width, 0,,,,5, equal to the lane width. In addition, we clip Oi,4, S Timax, Vi, t; and always input a fixed number of segments N₂ to our policy. More details and visualizations of the route representation are provided in the supplementary material.
Token Embeddings. Our model is illustrated in Fig. 2. As a first step, applying a transformer backbone requires the generation of embeddings for each input token, for which we define linear projection p R - HR (where is the desired hidden dimensionality). To obtain token em- beddings e,,,, add the projected input tokens o,, to a learnable object type embedding vector e, E R. e, E RH indicating to which type the token belongs (vehicle or route segment).
3

---

Explamaaility
Cuaypoints
Attention high
Luchioles
GRU
P
h.10
low
Attention
Transformer Encoder 7
Soene
Route
Object Type Emb. Token Emb. p(a.)
Linear Projection p
Igo
Tokenization
05,
Velicles V
Houle &
Figure 2: Planning Transformer (PlanT). We represent a scene (bottom left) using a set of objects containing the vehicles and route follow (green arrows). We embed these via linear projection (bottom right) and process them with a transformer encoder. PlanT outputs future waypoints with a GRU decoder. We use a self-supervised auxiliary task of predicting the future of other vehicles. Further, extracting and visualizing the attention weights yields an explainable decision (top left).
p((o))) e ((Oo))) +
Oi, E Vi. O2, E St.
e
Waypoint Prediction. The main building block for the IL policy is a standard trans- former encoder, T IR Ve+ + R VV++S++IXH based on the BERT architecture [37]. Specif- ically, define learnable [CLS] token, с E R (based token embeddings to obtain the transformer input [c, e1.1, ev,++ The [CLS] token's 's process- ing through involves an iitention-basee aggregation of the features from all other tokens, after which it is used for generating the waypoint predictions via an auto-regressive waypoint decoder, y: (#++)) - IR W x2 For a detailed description of the waypoint decoder architecture, see [18, 26]. We concatenate the binary traffic light flag, to the transformer output the initial hidden state to the decoder which makes use of GRUs [68] to predict the future trajectory W, of the ego vehi- cle, centered at the coordinate frame of the current time-step I. The trajectory is represented by sequence of 2D) waypoints in BEV space, {wi.- (Iw: yio) ) w=t+1 for W future time-steps:
Main Task:
[37, 39]) and stack this with other
We (le, h, (0)), where h, ([c,, 01,1, sv.+..
Auxiliary Task: Vehicle Future Prediction. In addition to the primary waypoint prediction task, propose the auxiliary task of predicting the future attributes of other vehicles. This is aligned with the overall driving goal in two ways. (1) The ability to reason about the future other vehicles important in urban environment as it heavily influences the ego vehicle's main task is to predict the ego vehicle's 's future trajectory. which means the output feature of the transformer needs encode all the information necessary predict the future. Supervising the outputs of all vehicles on similar task (i.e., predicting vehicle poses at a future time-step) exploits synergies between the task of the ego vehicle and the other vehicles [69, 30]. Specifically, using the output embeddings {hr corresponding to all vehicle tokens O,, V., we predict class 16
future. (2) Our
probabilities ([Pr,+1,0 1= i= i= for the speed, position, orientation, and extent attributes from the next time-step (0,,411, 16 using a linear layer per attribute type {wa : IR 16
Pa,++1a Softmax (h;.)), where a = 1,..., 6.
(3)
We choose discretize each attribute into Z, bins allow for uncertainty in the predictions since the future is multi-modal. This is also better aligned with how humans drive without predicting exact locations and velocities, where rough estimate is sufficient to make safe decision.
Loss Functions. Following recent driving models [9, 11, 30, 26), we leverage the In loss to the ground truth future waypoints w as our main training objective. For the auxiliary task, we calculate

---

the cross-entrope loss Lee using a one-hot encoded representation Pf, of the ground truth future vehicle attributes 1. We train the model in a multi-task setting using a weighted combination of these losses with a weighting factor
C
W
w
Lo (Pi,++,, P Plr+1a)
(4)
Coaypoints
Quehsoiee
4 Experiments
In this section, we describe our experimental setup, evaluate the driving performance of our ap- proach, analyze the explainability of its driving decisions, and finally discuss limitations.
Dataset and Benchmark. We use the expert, dataset and evaluation benchmark Longest 6 proposed by [26]. The expert policy is a rule-based algorithm with access to ground truth locations of the vehicles as well as privileged information that is not available to PlanT such as their actions and dynamics. Using this information, the expert determines the future position of all vehicles and estimates intersections between its own future position and those of the other vehicles to prevent most collisions. The dataset collected with this expert contains 228k frames. We use this as our reference point denoted by 1x. For our analysis, we also generate additional data following [26] but with different initializations of the traffic. The data quantities we use are always relative the original dataset (i.e., 2x contains double the data, 3x contains triple). We refer the reader to [26] for a detailed description of the expert algorithm and dataset collection.
Metrics. We report the established metrics of the CARLA leaderboard [70]: Route Completion (RC), Infraction Score (IS), and Driving Score (DS), which is the weighted average of the RC and IS. In addition, we show Collisions with Vehicles per kilometer (CV) and Inference Time (IT) for one forward pass of the model, measured in milliseconds on a single RTX 3080 GPU.
To highlight the advantages of learning-based plamning, we include a rule-based plan- ning baseline that uses the same inputs PlanT. It follows the same high-level algorithm as the expert but estimates the future of other vehicles using a constant speed assumption since it does not have access to their actions. AIM-BEV [18] is recent privileged agent trained using IL. It uses BEV semantic map input with channels for the road, lane markings, vehicles, and pedestrians, and GRU identical to PlanT to predict trajectory for the ego vehicle which is executed using lateral and longitudinal PID controllers. Roach [12] is a Reinforcement Learning (RL) based agent with similar input representation as AIM-BEV that directly outputs driving actions. Roach and AIM- BEV are the closest existing methods to PlanTT. However, they use a different input field of view in their representation leading to sub-optimal performance. We additionally build PlanCNN, a more competitive CNN-based approach for planning with the same training data and input information as PlanT, which adapted from AIM-BEV imput a rasterized version of our object-level represen- tation. We render the oriented vehicle bounding boxes in one channel, represent the speed of each pixel in a second channel, and render the oriented bounding boxes of the route in the third channel. We provide detailed descriptions of the baselines in the supplementary material.
Baselines.
Implementation. Our analysis includes three BERT encoder variants taken from [71]: MINI, SMALL. and MEDIUM with 11 11.2M, 28.8M and 41.4M parameters respectively. For PlanCNN, we experiment with two backbones: ResNet- 18 and ResNet-34. We choose these architectures to maintain an IT which enables real-time execution. We train the models from scratch on 4 RTX 2080Ti GPUs with a total batch size of 128. Optimization is done with Adam [72] for 47 epochs with an initial learning rate of 10-4 which we decay by 0.1 after 45 epochs. Training takes approximately 2.8, 3. 3.4, and hours for the three BERT variants on the 1x dataset. We the weight decay to 0.1 and clip the gradient norm at 1.0. For the auxiliary objective, we use quantization precisions Z, of 128 bins for the position, 4 bins for the speed and 32 bins for the orientation of the vehicles. We use T,, 0 and öt = for auxiliary supervision. The loss weight À is set to 0.2. By default, we use D maz -30 m, N, = 2, and Lmaz =10 m. For our experiment with full driving stack, we use a perception module based on TransFuser [26] to obtain the object-level input representation for PlanT. Additional details regarding this perception module as well as detailed ablation studies on the multi-task training and input representation hyperparameters are provided in the supplementary material.
5

---

Method
Input
DS f
RC
IS
CV
IT
Sensor
LAV= [30]
Camera LiDAR
32.74:145
70.3613.14
0.51
0.84=0.11
TransFuser [26]
Camera LiDAR
47.30:+572
93.38:1.20
0.50=2000
2.44
101.24
PlanT w/ perception
Camera LiDAR
57.661:5.1
88,201:0.94
0.65
0.97=0.09
37.61
Rule-based
Obj. + Route
38 38.00+1164
29.09+2.12
0.84:0000
0.64=0.07
Privileged
AIM-BEV [18]
Rast. Obj + HD Map
45.06::1.68
78.31+1112
0.55=001
1.67=0.16
18.14
Roach [12]
Rast. Obj. + HD Map
55.27
88.16:11.22
0.62:0002
0.76=0.07
3.24
PlanCNN
Rast. Obj. + Rast. Route
77.47:+1.44
94.53-+2.59
0.81
0.43=0005
28.94
PlanT
Obj. Route
81.36::6.4
93.55+:2.62
0.87=0005
0.31
10.79
Expert [26]
Obj. + Route Actions
76.91:225
88.67:20.6
0.862003
0.28
We show the mean for 3 evaluations. PlanT reaches expert-level performance and requires significantly less inference time than the baselines. *We evaluate the publicly available checkpoints at the time of submission for LAV and Roach. See supplementary material for results with checkpoints released later.
Table 1: Longestó Results.
4.1 Obtaining Expert-Level Driving Performance
In the following, we discuss the key findings of our study which enable expert-level driving with learned planners. We begin with a discussion of the privileged methods and analyze the sensor- based methods in Section 4.2. Unless otherwise specified, the experiments consider the largest version of our dataset (3x) and models (MEDIUM for PlanT, ResNet-34 for PlanCNN).
Input Representation. Table I compares the performance on the Longestó benchmark. The rule- based system acts cautiously and gets blocked often. Among the learning- based methods, both PlanCNN and PlanTT significantly outperform AIM-BEV [18] and Roach [12]. We systematically break down the factors leading to this in Table 2a by studying the following: (1) the representation
used for the road layout, (2) the horizontal field of view, (3) whether objects behind the ego vehicle are part of the representation, and (4) whether the input representation incorporates speed.
Roach uses the same view the sides AIM-BEV but additionally includes 8 m the back and multiple input frames to reason about speed. We see in Table 2a that training PlanCNN in configuration close to Roach (with the key differences being the removal of details from the map and a 0 m back view) results in a higher DS (59.97 vs. 55.27), demonstrating the importance of the route representation for urban driving. While additional information might be important when moving to more complex environments, our results suggest that the route is particularly important. Increasing the side view from 19.2 to 30 m improves PlanCNN from 59.97 to 70.72. Including vehicles to the rear further boosts PlanCNN's DS to 77.4 and improves PlanT's DS from 72.86 to 81.36. These results show that a full 360 field of view is helpful to handle certain situations encountered during our evaluation (e.g. changing lanes). Finally, removing the vehicle speed input significantly reduces the DS for both PlanCNN and PlanT (Table 2a), showing the importance of cues regarding motion.
Scaling, In Table 2b, we show the impact of scaling the dataset size and the model size for PlanT and PlanCNN. The circle size indicates the inference time (IT). First, we observe that PlanT demonstrates better data efficiency than PlanCNN, e.g., using the Ix data setting is sufficient to reach the same performance as PlanCNN with 2x. Interestingly, scaling the data from 1x to 3x leads to expert-level performance, showing the effectiveness of scaling. In fact, Plan'T (81.36) outperforms the expert (76.91) in some evaluation runs. We visualize one consistent failure mode of the expert that leads to this discrepancy in Fig. 3a. We observe that the expert sometimes stops once has already entered an intersection if it anticipates a collision, which then leads to collisions or blocked traffic. On the other hand, PlanT learns to wait further outside an intersection before enter- ing which is a smoother function than the discrete rule-based expert and subsequently avoids these infractions. Importantly, in final setting, PlanTwEDrUM is around 3x as fast as PlanCNN while being 4 points better in terms of the DS and Plan TINII is 5.3 x as fast (IT=5.46 ms) while reaching the same DS as PlanCNN. This shows that PlanT is suitable for systems where fast inference time is a requirement. We report results with multiple training seeds in the supplementary material.
Loss. A detailed study of the training strategy for PlanT can be found in the supplementary material, where we show that the auxiliary loss proposed in Eq. (4) is crucial to its performance. However, since this is self-supervised objective, it can be incorporated without additional annotation costs. This is in line with recent findings on training transformers that show the effectiveness of supervising multiple output tokens instead of just single [CLS] token [73].

---

Side (m)
Back (m)
Speed
Map
85
Method
DS
80
AIM-BEV [18]
V
19.2
1.60
75
Roach [12]
19.2
8
15.27
19.2
0
V
19.47
V
70.72+2.99
65
PlanCNN
30
V
77.47=+1.4
60
30
30
69.13:1143
Expert
PlanT-mini PlanT-small T-mni-medium
55
PIanCNN-16 PlanCNN-30 34
30
0
V
72.86::5.56
PlanT
30
V
81.36:16.54
30
12.34
1.0
1.5
2.0
2.5
3.0
Dataset
(a) Input Representation. DS on Longestó (3 eval- uation runs) with different input properties.
(b) Scaling. Mean DS Longestó for different dataset and model sizes. Circle size shows IT.
We investigate the choices of the input representation (Table 2a) and architecture (Table 2b). for learning-based planners. Including vehicles to the back of the ego, encoding vehicle speeds, and scaling to large models/datasets is crucial for the performance of both PlanCNN and PlanT.
Table 2:
4.2 Combining an Off-the-Shelf Perception Module with PlanT
Next, we discuss the results of the sensor-based methods in Table 1. We compare the proposed approach to LAV [30] and TransFuser [26], which are recent state-of-the-art sensor-based meth- ods. Our perception module is based on TransFuser, enabling fair comparison to this approach. Therefore, Plan'T with perception only detects vehicles
front and has limited view to the sides (16m instead of 30m). Our approach outperforms TransFuser [26] by 10.36 points and LAV by 24.92 points. While TransFuser uses an ensemble and manually designed heuristics to creep forward if stuck [26], these are unnecessary for PlanT with perception. Since we do not use an ensemble, we observe a 2.7x x speedup (101.24 ms vs. 37.61 ms) in IT compared to TransFuser. We refer the supplementary material for more detailed analysis.
4.3 Explainability: Identification of Most Relevant Objects
Finally, we investigate the explainability PlanT and PlanCNN by analyzing the objects in the scene that are relevant and crucial for the agent's decision. In particular, measure the relevance of an object in terms of the learned attention for PlanT and by considering the impact that the removal of each object has the output predictions for PlanCNN. To quantify the ability to reason about the most relevant objects, we propose a novel evaluation scheme together with the Relative Filtered Driving Score (RFDS). For the rule-based expert algorithm, collision avoidance depends on a single vehicle which it identifies as the reason for braking. To measure the RFDS of a learned planner, we run one forward pass of the planner (without executing the actions) to obtain a scalar relevance score for each vehicle in the scene. We then execute the expert algorithm while restricting its observations to the (single) vehicle with the highest relevance score. The RFDS is defined as the relative DS of this restricted version of the expert compared to the default version which checks for collisions against all vehicles. We describe the extraction of the relevance score for PlanT and PlanCNN in the following. Our protocol leads to a fair comparison of different agents the RFDS does not depend on the ability to drive itself but only on the obtained ranking of object relevance.
Baselines. As a naïve baseline, we consider the inverse distance the ego vehicle as vehicle's relevance score, such that the expert only sees the closest vehicle. For PlanT, we extract the relevance score by adding the attention weights of all layers and heads for the [CLS] token. This only requires a single forward pass of PlanT. Since PlanCNN does not use attention, we choose a masking method to find the most salient region in the image, using the same principle as [59, 58, 57]. remove one object at time from the input image and compute the Ln distance to the predicted waypoints for the full image. The objects are then ranked based on how much their absence affects the L1 distance.
Method
RFDS 1
Inverse Distance PlanCNN + Masking
29. 13
82.83=6.79
PlanT Attention
96.82#2.12
Table RFDS. Relative score of the expert when only observing the most relevant vehicle accord- ing the respective planner.
7

---

PlanT
PlanT
PlanCNN
Expert
l2
(a) PlanT vs. Expert. Plan'T waits further outside the intersection than the expert to avoid collision.
(b) REDS. Vehicles with the highest relevance score are marked with a red bounding rectangle. We show examples for successful matching of relevance score and intuition (green frames) and failures (red frames).
Figure 3:
We contrast failure case of the expert to PlanT (Fig. 3a) and show the quality of the relevance scores (Fig. 3b). The ego vehicle marked with a yellow triangle and vehicles that either lead to collisions
intuitively the most relevant in the scene are marked with blue box.
Results. We provide results for reasoning about relevant objects Table Both planners significantly outperform the distance-based baseline, with PlanT obtaining mean RFDS of 96.82 compared to 82.83 for PlanCNN. We show qualitative examples in Figure 3b where we highlight the vehicle with the highest relevance score using red bounding rectangle. Both planners correctly identify the most important object in simple scenarios. However, PlanT is also able to correctly identify the most important object in complex scenes. When merging into lane (examples 1 & 2 from the left) correctly looks at the moving vehicles coming from the rear to avoid collisions. Example 3 shows advanced reasoning about dynamics. The two vehicles closer the ego vehicle are moving away at a high speed and are therefore not as relevant. PlanT already pays attention the more distant vehicle behind them this is the one that it would collide with if it does brake. One of the failures of PlanTT we observe is that it sometimes allocates the highest attention a very close vehicle behind itself (example 4) and misses the relevant object. PlanCNN has more prominent errors when there are a large number of vehicles in the scene or when merging lanes (examples 2 & 3). To better assess the driving performance and relevance scores we provide additional results in the supplementary video.
5 Conclusion
In this work, take step towards efficient, gigh-performane, explainable planning for tonomous driving with a novel object-level representation and transformer-basee architecture called PlanT. Our experiments highlight the importance of correctly encoding the ego vehicle's route for planning. We show that incorporating a 360 field of view, information about vehicle speeds, and scaling up both the architecture and dataset size of learned planner essential achieve state- of-the-art results. Additionally, Plan'T significantly outperforms state-of-thh-- end-to-end sensor- based models even with noisy and incomplete input representation obtained via perception mod- ule. Finally, we demonstrate that PlanT can reliably identify the most relevant object in the scene via a new metric and evaluation protocol that measure explainability.
Firstly, the expert driver used in our IL-based training strategy does not achieve a perfect score (Table and has certain consistent failure modes (Fig. 3a, more examples in the supplementary material). Human data collection to address this would be time-consuming (the 3x dataset used in experiments contains around 95 hours of driving). Second, all experiments are conducted in simulation. Real-world scenarios are more diverse and challenging. However, CARLA is high-fidelity simulator actively used by many researchers for autonomous driving, and previous findings demonstrate that systems developed in simulators like CARLA can be transferred the real world [31, 74, 75], Finally, our experiment with perception (Section 4.2) uses single off- the-shelf perception module that was not specifically optimized for PlanT, leading to sub-optimal performance.
Limitations.
well-known limitation of modular systems that downstream modules cannot recover easily from errors made by earlier modules. A thorough analysis of perception robustness and uncertainty encoding are important research directions beyond the scope of this work.

---

Acknowledgments
This work was supported by the BMWi (KI Delta Learning, project number: 19A190130), the BMBF (Tübingen AI Center, FKZ: 011S 011S18039A), the DFG (SFB 1233, TP 17, project number: 276693517), by the ERC (853489 DEXIM), and by EXC (number 2064/1 project number 390727645). We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting K. Renz, K. Chitta and O.-B. Mercea. The authors also thank Niklas Hanselmann and Markus Flicke for proofreading and Bernhard Jaeger for helpful discussions.
References
[1] C. Thorpe. M. H. Hebert, T. Kanade, and S. A. Shafer. Vision and navigation for the carnegie- mellon navlab. PAMI, 10(3):362-372, May 1988.
[2] E. D. Dickmanns, R. Behringer, D. Diekmanns, T. Hildebrandt, M. Maurer, F. Thomanek, and Schiehlen. The seeing passenger car 'vamors-p'. In /V, 1994.
[3] J. Leonard, P. How, S. J. Teller, M. Berger, S. Campbell, G. A. Fiore, L. Fletcher, E. Fraz- zoli, A. S. Huang. S. Karaman, O. Koch, Y. Kuwata, D. Moore, E. Olson, S. Peters, J. Teo, R. Truax, M. R. Walter, D. Barrett, A. Epstein, K. Maheloni, K. Moyer, T. Jones, R. Buckley,
M. E. Antone, R. Galejs, S. Krishnamurthy, and J. Williams. A perception-driven autonomous urban vehicle. JFR, 25(10):27-777, 2008.
[4] W. Xu, J. Pan, Wei, and J. M. Dolan. Motion planning under uncertainty for on-road au- tonomous driving. In ICRA, 2014.
[5] B. Paden, M. Cáp. S. Z. Yong, D. Yershov, and E. Frazzoli. A survey motion planning and control techniques for self-driving urban vehicles. IEEE Transactions Intelligent Vehicles, 2016.
[6] H. Fan, F. Zhu, C. Liu, I. Zhang. I. Zhuang, D. Li, W. Zhu, J. Hu, H. HiL, and Q. Kong. Baidu apollo EM motion planner. arXiv.org, 1807 2018.
[7] A. Sadat, S. Casas, M. Ren, X. Wu, Dhawan, and R. Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In ECCV, 2020.
[8] A. Cui, A. Sadat, S. Casas, R. Liao, and R. Urtasun. Lookout: Diverse multi- prediction and planning for self-driving. In ICCV, 2021.
[9] D. Chen, B. Zhou, V. Koltun, and Krähenbühl. Learning by cheating. In CoRL., 2019.
[10] W. Zeng, Luo, S. Suo, A. Sadat, B. Yang, Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
[11] A. Prakash, K. Chitta, and A. Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. In CVPR, 2021.
[12] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021.
[13] O. Scheel, I.. Bergamini, M. Wolezyk, B. Osiáski, and P. Ondruska. Urban driver: Learning drive from real-world demonstrations using policy gradients. In CoRL, 2021.
[14] M. Vitelli, Y. Chang, Y. Ye, M. Wolczyk, B. Osiński, M. Niendorf, H. Grimmett, Q. Huang, A. Jain, and P. Ondruska. Safetynet: Safe planning for real-world self-driving vehicles using machine-leareed policies. arXiv.org. 2109. 13602, 2021.
[15] D. Marr. Vision: A computational investigation into the human representation and processing of visual information. W. H. Freeman, San Francisco, 1982.
[16] E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental Science, 10(1):89-99) 96, 2007.
[17] P. Johnson. Object perception. In Oxford Research Encyclopedia of Psychology. Oxford University Press, 2018

---

[18] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generat- ing safety-critical driving scenarios for robust imitation via kinematics gradients. arXiv.org, 2204.13683 2022.
[19] K. Chitta, A. Prakash, and A. Geiger. Neat: Neural attention fields for end-to-end autonomous driving. In ICCV, 2021.
[20] A. Saha, O. Mendez Maldonado, C. Russell, and R. Bowden. Translating Images into Maps. ICRA, 2022.
[21] B. Zhou and P. Krähenbühl. Cross-view Transformers for real-time Map-view Semantic Seg- mentation. In CVPR, 2022.
[22] L. Peng, Z. Chen, Z. Fu, P. Liang, and E. Cheng. BEVSegFormer, Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs. arXiv.org, 2203.04056 2022.
[23] E. Xie. Z. Yu, D. Zhou, J. Philion, A. Anandkumar, S. Fidler, P. Luo, and J. M. Alvarez. MBBEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation. arXiv.org, 2204.05088, 2022.
[24] Z. Li. W. Wang. H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and Dai. BEVFormer: Learning B-rd's-Eye- View Representation from Multi-Camert Images via Spatiotemporal Transformers. arXiv.org, 2203. 17270, 2022.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, I.. Kaiser, and I. Polo- sukhin. Attention is you need. In NeurlPS, pages 5998 6008, 2017.
[26] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. PAMI, 2022.
[27] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and Koltun. CARLA: An open urban driving simulator. In CoR1,2017
[28] C. Chen, A. Seff, A. L. Kornhauser, and J. Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In ICCV, pages 2722-2730, 2015.
[29] F. Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving. ICCV, 2019.
[30] D. Chen and P. Krähenbühl. Learning from all vehicles. In CVPR, 2022.
[31] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun. Driving policy transfer via modularity and abstraction. In CoRL, 2018.
[32] A. Sax, B. Emi, A. R. Zamir. L. J. Guibas, S. Savarese, and J. Malik. Learning to navigate using mid-level visual priors. CoRL, 2019.
[33] A. Mousavian, A. Toshev, M. Fiser, J. Kosecká, A. Wahid, and J. Davidson. Visual represen- tations for semantic target driven navigation. In ICRA, 2019.
[34] B. Zhou, Krähenbühl, and V. Koltun. Does computer vision matter for action? Science Robotics, 4(30), 2019.
[35] A. Behl, K. Chitta, A. Prakash, E. Ohn-Bar, and A. Geiger. Label efficient visual abstractions for autonomous driving. In IROS, 2020.
[36] D. Wang, C. Devin, Q. Cai, P. Krähenbühl, and T. Darrell. Monocular plan view networks for autonomous driving. IROS, 2019.
[37] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.org. 2018.
[38] A. Radford, J. Wu, R. Child, Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
10

---

[39] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghami, M. Minderer, G. Heigold. S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x words: Transformers for image recognition at scale. arXiv.org, 2010.11929, 2020.
[40] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurIPS, 2021.
[41] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021.
[42] H. Furuta, Y. Matsuo, and S. S. Gu. Generalized decision transformer for offline hindsight information matching. arXiv.org, 2111.10364, 2021.
[43] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. arXiv.org, 2202.05607, 2022.
[44] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, T. Springenberg, et al. A generalist agent. arXiv.org. 2205.06175 2022.
[45] N. Djuric, V. Radosavljevic, H. Cui, T. Nguyen, F.-C. Chou, T.-H. Lin, and Schneider. Uncertainty-wware short-term motion prediction of traffic actors for autonomous driving. In WACV, 2020.
[46] S. Casas, W. Luo, and R. Urtasun. Intentnet: Learning to predict intention from raw sensor data. In CoRL, 2018.
[47] Hong. Sapp, and Philbin. Rules of the road: Predicting driving behavior with a convo- lutional model of semantic interactions. In CVPR, 2019.
[48] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. arXiv.org, 1910.05449, 2019.
[49] Y. Biktairov, M. Stebelev, I. Rudenko, O. Shliazhko, and B. Yangel. Prank: motion prediction based on ranking. In NeurIPS, 2020.
[50] T. Phan-Minh, E. C. Grigore, A. Boulton, O. Beijbom, and E. M. Wolff Covernet: Multi- modal behavior prediction using trajectory sets. In CVPR, 2020.
[51] Gao, C. Sun, H. Zhao, Y. Shen, Anguelov, C. Li. and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, June 2020.
[52] L. L. Li, B. Yang, M. Liang. W. Zeng, M. Ren, S. Segal, and R. Urtasun. End-to-end contextual perception and prediction with interaction transformer. In IROS, 2020.
[53] J. Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. J. Weiss, B. Sapp. Z. Chen, and J. Shlens. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR, 2022.
[54] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers. In CVPR, June 2021.
[55] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618- 626, 2017.
[56] P. Dabkowski and Y. Gal. Real time image saliency for black box classifiers. In NeurIPS, 2017.
[57] R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In ICCV, 2017.
[58] V. Petsiuk, Das, and K. Saenko. Rise: Randomized input sampling for explanation of black-box models. In BMVC, 2018.

---

[59] R. Fong, M. Patrick, and A. Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. ICCV, 2019.
[60] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
[61] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
[62] Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata. Textual explanations for self-driving vehicles. In ECCV, 2018.
[63] Kim and J. Canny. Interpretable learning for self-driving cars by visualizing causal attention. In ICCV, 2017.
[64] R. McAllister, B. Wulfe, J. Mercat, L. Ellis, S. Levine, and A. Gaidon. Control-aware predic- tion objectives for autonomous driving. ICRA, 2022.
[65] X. Zhou, D. Wang, and P. Krähenbühl. Objects as points. arXivorg, 1904.07850, 2019.
[66] U. Ramer. An iterative procedure for the polygonal approximation of plane curves. Comput. Graph. Image Process., 1:244-256 256, 1972.
[67] D. H. Douglas and T. K. Peucker. Algorithms for the reduction of the number of points required to represent digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization, 10:112-122 1973.
[68] K. Cho, van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Ben- gio. Learning phrase representations using RNN encoder-decoder for statistical machine trans- lation. In Proc. of the Conference on Empirical Methods Natural Language Processing (EMNLP), 2014.
[69] J. Zhang and E. Ohn-Bar. Learning by watching. In CVPR, June 2021.
[70] Carla autonomous driving leaderboard. https///eaderboardd.aaa..or 2020.
[71] Ture, M.-W. Chang, K. Lee, and K. Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv.org. 1908.08962, 2019.
[72] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv.rr, 1711.05101, 2017.
[73] K. He, X. Chen, S. Xie, Li, Dollár, and R. B. Girshick. Masked autoencoders are scalable vision learners. arXiv.org, 2111.06377, 2021.
[74] D. Wang, C. Devin, Q. Cai, F. Yu, and T. Darrell. Deep object centric policies for autonomous driving. In ICRA, 2019.
[75] I. Gog, S. Kalra, P. Schafhalter, M. A. Wright, J. E. Gonzalez, and I. Stoica. Pylot: A modular platform for exploring latercy-accuracy tradeoff's in autonomous vehicles. In ICRA, 2021.
12


Reviews:
[{"review_id": "kBRZSe15Vls7", "reviewer": "Reviewer_LmXW", "summary": "This paper designs a novel object-level scene representation and proposes a transformer based framework to predict the set of waypoints given the route. The framework outperforms other baselines and is more efficient. The attention mechanism also improves the explain explainability of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.Transform the scene representation from pixel level to object level which is closer to human perception.\n2.Regarding the vehicles and routes as a sequence and utilize transformer to predict the waypoints, which is a novel and reasonable combination.\n3.The visualization of the attention score over other vehicles improves the explainability of the decision making process.\n4.The attached videos clearly demonstrates the effectiveness of the framework.\n\nWeaknesses:\nThe framework only considers the controlling process but ignore the perception process, which greatly simplifies the problem. I hope that the future work would take perception into consideration and construct a complete framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper designs a novel object-level scene representation and proposes a transformer based framework to predict the set of waypoints given the route. The authors transform the scene representation from pixel level to object level which is closer to human perception by regarding the vehicles and routes as a sequence and utilize transformer to predict the waypoints, which is a novel and reasonable combination.  The framework outperforms other baselines and is more efficient. The attention mechanism also improves the explain explainability of the method. However, the framework only considers the controlling process but ignore the perception process, which greatly simplifies the problem.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Some of the training details like the illustration of the input representations and training strategy should be replaced to the body part.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "k4XIJDyPWH1", "reviewer": "Reviewer_QCRj", "summary": "This paper presents PlanT, a transformer architecture for motion planning. Unlike most models which reason according to rasterized BEV scene representations, PlanT leverages an object-centric input representation and simple route encoding. The model is trained similar to prior works using behavior cloning with supervision obtained from a rules-based expert with access to privileged information. The authors claim PlanT's architecture and input representation leads to improved planning performance and more explainable outputs, since the transformer's attention weights provide a measure of importance of each object in the scene. The authors provide a set of comprehensive experiments in the CARLA simulator to support these claims. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- **Clear presentation:** The paper is very well written and easy to follow.\n- **Very thorough experiments:** I really appreciate the scientific rigour in the experimental analysis and how the authors control for each change between the models. I appreciate the authors comparing against PlanCNN and found the analysis in Table (2a) very well done as it shows exactly where the performance improvements of the model come from. \n- **Simple yet effective model:** The proposed model is quite simple, yet is convincingly shown to be the best performing model. The qualitative difference between the explainability visualization of PlanCNN and PlanT shown in the video is apparent and helps support the claim that PlanT is a more explainable model. \n\n**Weaknesses**\n- **Assumes perfect perception**: As mentioned in the limitations, this paper only considers the problem of planning and assumes perfect perception. While I understand the desire to reduce the scope of the problem, I believe the experimental section or supplementary should show at least one experiment with end-to-end results (from sensor inputs). This would help provide a sense of what the gap is between using GT inputs vs. noisy perception.  \n- **Simplified route representation:** While it was an interesting contribution to the paper that a simplified route is sufficient for state-of-the-art performance, this seems like a limitation for a real-world setting. How would this representation support a discretionary lane change (one that isn't needed to follow a route, but that the planner performs for a safer and/or faster driving)? What is the intuition for why all lane information is not helpful for better prediction of the other actors?  \n- **Missing connection to transformers for motion-forecasting:** While the end objective is clearly different, the problems of motion forecasting and motion planning share many similarities. Therefore, I think it could be beneficial to mention the connections to the many recent works on leveraging transformers for motion-forecasting and highlight the differences in this architecture. I find that the proposed architecture has similarities with SceneTransformer [1], which also leverages a Transformer to predict the future states of each actor - I believe the key differences between the two are the end objective, the architectural details (cross-attention) and the route representation used in this paper. Additionally, past works in transformers for motion-forecasting [2] have also visualized attention weights to understand the significance actors have on each other (albeit for the motion forecasting task), which may be worth mentioning in the related works. \n- **Only tested in simulation and reliance on a rules-based expert:** This was also mentioned in the paper's limitations, but I'm curious how the authors would imagine using this system in the real-world? Since the model is data hungry, it seems like it would be most feasible to train in simulation and try to transfer the model to the real-world? If that is the case, why leverage an expert and not consider other training schemes like RL? When using the same field of view, it seems like Roach and PlanCNN are within performance (accounting for the confidence intervals). Do you think this model could be trained or fine-tuned with RL in the future to outperform the expert even further? \n\nReferences: \n- [1] Ngiam et al., Scene Transformer: A unified architecture for predicting multiple agent trajectories, ICLR 2022\n- [2] Li et al., End-to-end Contextual Perception and Prediction with Interaction Transformer, IROS 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am on the fence for this paper but leaning towards accept. On the one hand, I think this method has limited novelty (when considered in the context of recent transformers for motion forecasting) and has several limitations (most of which are  mentioned in the paper's limitations section) which would need to be addressed before it could be deployed in a real-world setting. However, many of these simplifying assumptions are common in the research community and I find the experimental analysis of this paper very thorough and the detailed ablations and experiments insightful. Additionally, while there are similarities with prior work, applying the transformer to object-centric representations for motion planning is novel, as far as I know. Therefore, since the technical quality is high, the experiments are insightful and the contributions could provide value to the community, I would lean towards acceptance. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "See weaknesses section. \n\nSome additional minor comments: \n- I personally find the title of this paper misleading. I think it would be more accurate to include \"transformer\" and \"object-centric\" as those highlight the key contributions of the paper. \n- Re the auxiliary prediction loss the authors say, \"However, since this is a self-supervised objective, it can be incorporated without additional annotation costs\". I find this statement misleading as it is only true in the simulated setting and may not necessarily be true in a real-world setting. It's not clear for real-world data whether one would use the outputs from the perception system (which would be noisy, but not require annotation) or GT human labels (which would be more accurate, but costly to annotate). "}}, {"review_id": "NMkOfKv4B_Z", "reviewer": "Reviewer_H8PS", "summary": "This paper presents a transformer-based approach for planning (PlanT) in the context of autonomous driving that takes as input a sparse object-based representation of the environment (in contrast to the denser HD maps commonly used in practice). The approach is demonstrated in challenging simulated autonomous driving scenarios and surpasses prior methods (many based on denser input representations) across several benchmark driving metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The technical approach, as I understand it, seems sound. The idea of using a purely object-based representation as input to a learned planner seems neat, and the ability to assign some scores to objects related to their \"relevance\" seems interesting.\n\n- The fact that the planner here takes a *sparse* representation is particularly interesting, as this requires significantly less data than HD maps currently in use (making it also potentially easier to process at high rates).\n\n- The supplemental video was **very** helpful in understanding the system and visualizing the output.\n\n- The experimental evaluation is very thorough, and the detailed appendices are appreciated.\n\n# Weaknesses\n(I am combining weaknesses here with issues/clarifications)\n\n- The technical exposition in Sec 3 is very dense. I believe this is in part due to the fact that essentially the entire system is described verbally. I could see this making it difficult for a practitioner to attempt to reproduce this method without access to the code. I would encourage the authors to try to define concisely (even at a high-level) the mathematical model captured in part by Figure 1. In particular, with the current presentation, it is tremendously difficult to visualize how the verbally described components fit together (even with the aid of Fig. 2), or determine any sort of relative importance (i.e. what are really the \"main\" ideas versus auxiliary components). It seems that almost everything is essentially given the same \"priority\" in the current presentation. In the absence of such a mathematical model, it would help to give an overview figure, perhaps one similar to Fig. 2, but to provide references to the sections in text where components are discussed. The pair of eq (1) with Fig 2 is a great example of this working successfully. I particularly appreciated that the different loss terms were cross references, and the choice of notation here helped convey that. I wonder if a similar approach could be taken for the rest of the system?\n\n- Strictly speaking, is it fair to call the output of the system \"explainable\"? I am not an expert in explainable AI, but while the object attention weights visualized in the supplemental video for PlanT do look more \"stable\" than PlanCNN, it was rarely obvious to me why PlanT highlighted particular car as \"relevant\" (except in the event of the obvious interpretation that the most relevant car was simply the one immediately in front of the agent). Independent of this, the ability to provide these attention weights seems interesting to me, and does (I believe) help visualize or interpret the internals of the model, compared to a black-box model.\n\n- The downside of the sparseness of the object-based representation in practice is that it may be more error prone than an HD map, for the simple reason that consistency of the (denser) information encoded in the HD map can be used to reason about e.g. outliers. Another natural issue is that the object-based map will typically rely crucially on the output of learned perception models, which can produce spurious detections or misclassifications. The limitations section already mentions that object states were extracted from the simulator directly (and it is reasonable in this work to focus on planning specifically), however, I would suggest discussing the issue of robustness, particularly with respect to spurious or missed object detections and misclassifications. In principle, this may be something that could be tested in simulation.\n\n- Minor, but one of the stated contributions is that \"we demonstrate that a simple object-level representation is sufficient to encode all the information relevant for planning in urban driving environments.\" This seems like far too broad a claim to be made without more thorough evaluation in real environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The contribution here (a transformer-based planning approach that processes a parsimonious object-based representation for autonomous driving) seems relevant and novel to me. The paper overall is good, but there are some clarity issues with the technical presentation; this is where it could be improved the most, in my opinion. The evaluation is thorough and clear.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "See Weaknesses (reproduced here)\n\n- The technical exposition in Sec 3 is very dense. I believe this is in part due to the fact that essentially the entire system is described verbally. I could see this making it difficult for a practitioner to attempt to reproduce this method without access to the code. I would encourage the authors to try to define concisely (even at a high-level) the mathematical model captured in part by Figure 1. In particular, with the current presentation, it is tremendously difficult to visualize how the verbally described components fit together (even with the aid of Fig. 2), or determine any sort of relative importance (i.e. what are really the \"main\" ideas versus auxiliary components). It seems that almost everything is essentially given the same \"priority\" in the current presentation. In the absence of such a mathematical model, it would help to give an overview figure, perhaps one similar to Fig. 2, but to provide references to the sections in text where components are discussed. The pair of eq (1) with Fig 2 is a great example of this working successfully. I particularly appreciated that the different loss terms were cross references, and the choice of notation here helped convey that. I wonder if a similar approach could be taken for the rest of the system?\n\n- Strictly speaking, is it fair to call the output of the system \"explainable\"? I am not an expert in explainable AI, but while the object attention weights visualized in the supplemental video for PlanT do look more \"stable\" than PlanCNN, it was rarely obvious to me why PlanT highlighted particular car as \"relevant\" (except in the event of the obvious interpretation that the most relevant car was simply the one immediately in front of the agent). Independent of this, the ability to provide these attention weights seems interesting to me, and does (I believe) help visualize or interpret the internals of the model, compared to a black-box model.\n\n- The downside of the sparseness of the object-based representation in practice is that it may be more error prone than an HD map, for the simple reason that consistency of the (denser) information encoded in the HD map can be used to reason about e.g. outliers. Another natural issue is that the object-based map will typically rely crucially on the output of learned perception models, which can produce spurious detections or misclassifications. The limitations section already mentions that object states were extracted from the simulator directly (and it is reasonable in this work to focus on planning specifically), however, I would suggest discussing the issue of robustness, particularly with respect to spurious or missed object detections and misclassifications. In principle, this may be something that could be tested in simulation.\n\n- Minor, but one of the stated contributions is that \"we demonstrate that a simple object-level representation is sufficient to encode all the information relevant for planning in urban driving environments.\" This seems like far too broad a claim to be made without more thorough evaluation in real environments."}}, {"review_id": "Iq5Anjl-v6Z", "reviewer": "Reviewer_ZSWX", "summary": "The paper proposes PlanT, a transformer-based end-to-end planner that uses object-level representations. The idea is very simple, the transformer takes vehicle bounding boxes as well as a discretized route as input tokens and outputs the planned future waypoints. Additionally, they predict the future of other vehicles in the scene as an auxiliary task that is shown to be helpful for better representation learning. The contributions are the method itself as well as a metric to measure the explainability of the planner.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Paper is written clearly\n- The motivation for sparse representations outlined in the abstract is fair.\n- Method is simple and easy to understand\n- The proposed metric to evaluate the explainability of a planner, RFDS, is interesting. However, I believe the usefulness of the metric to drive progress in the field is questionable, as it only gets at the attention placed on other objects.\n\n### Weaknesses\n- **Main weakness:** the paper claims as a contribution that they demonstrate that \"a simple object-level representation is sufficient to encode all the information relevant for planning in urban driving environments\". This claim is false, and the reason is that the proposed method assumes perfect perception. While this limitation is recognized in the limitations section, its importance is well beyond what the paper assumes. The reason why other papers use BEV dense representations is to *capture uncertainty*. When not having perfect perception and dealing with challenges such as occlusion, uncertainty on the current and future states of objects grows very significantly. Since the proposed method accesses the internal state of the simulator to know the object locations, speed, etc., there is no uncertainty in the current states of objects, and very little on future ones since CARLA's actors are overly simplistic. Since there is no uncertainty about object existence, all the information can be captured at the object level and thus the proposed method work best. However, this is not a realistic robotic application. Even if the authors would do experiments with a real object detector, CARLA is not the most suitable environment for that as perception is much easier than in the real-world.\n- The paper also claims to be 5.3x faster than previous method. First, that is not obvious from Table 1. Moreover, even if the claim were true with the current experiments, we come back to the fact that the method doesn't perform object detection. Running object detection would impact inference time in two ways. First, the time to run the object detector. Second, the number of actors would increase significantly to attain an acceptable recall, further increasing runtime.\n- It is also claimed that route is all the map representation needed. Again, I believe this is true for their experiments, but an flawed argument in reality. First, CARLA's scenarios are very simple and the other actors quite reactive. This makes it easy for the SDV to never have to steer off the route to perform mitigation maneuvers like a safe stop. Second, we come back to assuming perfect perception. Since the method has a privileged understanding of the scene there is much less uncertainty, which translates into the SDV never having to steer off the route.\n- While the attention from SDV token to other actor token is interesting as an interpretable proxy for importance, this isn't really novel and was previously proposed by CAPO (see citation below).\n- It is not fully clear what is the output representation for the auxiliary task of predicting the future trajectories of other agents. As a consequence, $\\mathcal{L}_{CE}$ is also unclear.\n```\n@inproceedings{mcallister2022control,\n  title={Control-Aware Prediction Objectives for Autonomous Driving},\n  author={Rowan McAllister and Blake Wulfe and Jean Mercat and Logan Ellis and Sergey Levine and Adrien Gaidon},\n  booktitle={International Conference on Robotics and Automation (ICRA)},\n  year={2022}\n}\n```", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend rejection for this paper given that the claims are not validated by the method and experimental setup. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "Please see weaknesses section. I recommend that the authors add a perception module, repeat their experiments, revisit the claims and resubmit to a subsequent conference. While I believe the work is interesting and it is worth continue building on top of, with the experiments as they stand I don't believe readers can take any valuable conclusion."}}]

Author rebuttal:

Decision: Accept (poster)