Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
Authors: Penghao Wu
Xiaosong Jia
Li Chen
Junchi Yan
Hongyang Li
Yu Qiao
Venue: NeurIPS 2022 Accept (2022)
Keywords: End-to-end Autonomous Driving

Abstract: Current end-to-end autonomous driving methods either run a controller based on a planned trajectory or perform control prediction directly, which have spanned two separately studied lines of research. Seeing their potential mutual benefits to each other, this paper takes the initiative to explore the combination of these two well-developed worlds. Specifically, our integrated approach has two branches for trajectory planning and direct control, respectively. The trajectory branch predicts the future trajectory, while the control branch involves a novel multi-step prediction scheme such that the relationship between current actions and future states can be reasoned. The two branches are connected so that the control branch receives corresponding guidance from the trajectory branch at each time step. The outputs from two branches are then fused to achieve complementary advantages. Our results are evaluated in the closed-loop urban driving setting with challenging scenarios using the CARLA simulator. Even with a monocular camera input, the proposed approach ranks first on the official CARLA Leaderboard, outperforming other complex candidates with multiple sensors or fusion mechanisms by a large margin. The source
code is publicly available at https://github.com/OpenPerceptionX/TCP

Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
Penghao Wu Shanghai AI Laboratory Shanghai Jiao Tong University
Xiaosong Jia"
Li Chen"
Shanghai Jiao Tong University Shanghai AI Laboratory j jiaziaosong$s ltres//sttuuuu......
Shanghai Al Laboratory jichen@pjlab.. org. en
mupenghaocraigãs edu. cn
Junchi Yan
Hongyang Li
Yu Qiao
Shanghai Jiao Tong University Shanghai AI Laboratory
Shanghai AI Laboratory Shanghai Jiao Tong University lihongyang@pjlab. org.
Shanghai AI Laboratory qiaoyu@pjlab org.
janjunch@ssjuu edu. cn
Abstract
Current end-to-end autonomous driving methods either run a controller based on planned trajectory perform control prediction directly, which have spanned two separately studied lines of research. Seeing their potential mutual benefits to each other, this paper takes the initiative explore the combination of these well-developed worlds. Specifically, our integrated approach has two branches for trajectory planning and direct control, respectively. The trajectory branch predicts the future trajectory, while the control branch involves novel multi-step prediction scheme such that the relationship between current actions and future states be reasoned. The two branches are connected so that the control branch receives corre- sponding guidance from the trajectory branch each time step. The outputs from two branches are then fused to achieve complementary advantages. Our results are evaluated in the closed-loop urban driving setting with challenging scenarios using the CARLA simulator. Even with monocular camera input, the proposed approach ranks first on the official CARLA Leaderboard, outperforming other complex can- didates with multiple sensors or fusion mechanisms by large margin. The source code is publicly available https: //g thub. com//pennerrcptiiox//cc.
Introduction
End-to-end autonomous driving methods, which directly map raw sensor data to a planned trajectory low-level control actions, show the virtue of simplicity, conceptually avoiding the cascading error of complex modular design and heavy hand-crafted rules. The output prediction of the model for end-to-
end autonomous driving generally falls into two forms: trajectory/waypointss [48, 11, 46, 15, 29, 10] and direct control actions [17, 39, 18, 42, 12. 60, 9]. However, there is still no clear conclusion as to which of these two forms is better for all circumstances or certain scenarios.
Different from control predictions that could be directly applied to the vehicle, for methods that plan trajectory, additional controllers such PID controllers usually needed subsequent step to convert the planned trajectory into control signals. One attractive and potential supremacy of trajectory-based prediction is that it actually considers relatively longer time horizon into the future and could be further combined with other modules (e.g., multi-agent trajectory prediction [59, 10],
Equal Contribution. Work done when PW and XJ were interns at Shanghai Al Laboratory. Correspondence author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ct:
Success
Cdl:
Collision
Traj:
Off-Road
Traji
Success
Figure 1: Typical failure cases of two prediction paradigms. Red dots indicate the trajectory prediction. blue dots are the actual path following the red trajectory with PID controllers, and green dots denote the actual path from control-basee method. (a) trajectory- baased methods may struggle for big turns. (b) coatrol-based methods may have a reaction latency and suffer from abrupt obstacles due to focusing on current time step only. These observations motivate us to propose unified framework to combine these two worlds for mutual benefits.
semantic or occupancy prediction modules [15, 25]) reduce possible collisions. However, turning the trajectory into control actions so that the vehicle could follow the planned trajectory is not trivial [57]. The industry usually adopts sophisticated control algorithms such model predictive control performance [5, 21]. Simple PID controllers may perform worse in situations such as taking a big turn or starting the red light due to the inertial problem of end-to-end models [29]. For control- based methods, the control signals are directly optimized. Nevertheless, their focus on the current step may cause deferred reactions to avoid potential collisions with other moving agents. The independence between the control predictions of different steps also makes the actions of the vehicle more unstable or discontinuous. Fig. 1 shows two typical cases where two paradigms fail respectively. How to combine these two forms of prediction model as well as their outputs is an interesting yet relatively rarely studied area, which motivates this work.
to achieve reliable saajccory-fooloowe
One straightforward (but in fact rarely studied in literature) idea is to train control prediction model and a trajectory planning model separately, and combine their ultimate outputs directly. It can be viewed as an ensemble of two different models. However, such naive approach not only doubles the size of the model, but also ignores possible useful correlations between these two forms. To this end, we introduce the TCP (Trajectory-guidee Control Prediction) framework, packing these two branches into a unified framework. It can be viewed as a multi-task learning (MTL) [7, 2] framework where a shared backbone extracts common features with decreased computational complexity as well as the increased ability of generalization due to the close relationship between the two tasks [38, 34, 13]. Furthermore, to address the drawbacks of current control prediction methods, we delicately devise novel multi-step control branch and a trajectory-guice
control prediction scheme.
While trajectory planning considers several steps into the future, directly learning the control in a behavior cloning fashion [44, 41, 17, 18, 11] often focuses on the current time step only, given prior on each state-action pair as independent and identical distributed (IID). This assumption is not accurate and may hamper the long-term performance since the driving task is a sequential decision- making problem. To alleviate the problem, we propose to predict multi-step control actions into the future. However, the multi-step control process needs interactions with the environment. Thus we formulate a temporal module to learn the forward process and interactions between the ego agent and the environment. A temporal module implemented with GRU [16] progressively deals with the feature representation for each time step, implicitly taking into account the dynamic motion of agents, interaction among them and dynamic environment information such as the changing of traffic lights.
Additionally, to generate accurate control signals in the multi-step prediction scheme, the model should retrieve proper location information from current sensor input for different future time steps. For example, an agent may pay more attention to nearby regions for few early future time steps and far away regions for the remote ones. Considering that the knowledge has already been partly encoded in the trajectory branch, we adopt the attention mechanism to locate those critical and helpful areas in the long-term trajectory prediction branch, and guide the control prediction branch to pay attention to them at each future step in a corresponding way. As result, our model is capable of reasoning about how to optimize current control prediction so that the future states are similar to those from the expert when the predicted control actions are applied.

---

With the predicted trajectory and control signals from two branches, we propose situation based fusion scheme adaptively combine these two forms in self-ensemble way form the ultimate output according to the experiments results and prior knowledge. It combines the best of these two forms, which further boosts the performance under different scenarios.
TCP has shown superior performance when being validated in the CARLA driving simulator [20]. Our method, which only uses monocular camera, achieves a 75.137 driving score and ranks 1st on the public CARLA Leaderboard [1], even surpassing prior state-of-the-ar methods using multiple cameras and a LiDAR by 13.291 points. The main contributions of this paper include:
We examine two dominant paradigms for end-to-end autonomous driving: trajectory plan- ning and direct control, and propose to combine them in an integrated learning pipeline. To our knowledge, this is the first time that such two branches are jointly learned and fused for prediction.
A multi-step control prediction branch with temporal module and trajectory- at- tention is devised to enable temporal reasoning. To combine the best of two branches, we design a situation based scheme to fuse the two outputs.
As a simple yet strong baseline, our method with only monocular camera as input achieves new state-of-the-ar sensors. We conduct thorough ablation studies to verify the effectiveness of our approach.
the CARLA Leaderboard with many competitors using multiple
2 Related Work
2.1 End-to-end Autonomous Driving
Learning-basee end-to-end autonomous driving has emerged as an active research topic in recent years. Studies usually fall into two categories: reinforcement learning (RL) and imitation learning, RL a promising way to address the problem of being more robust to the distribution shifts of datasets. Liang et al. [39] use DDPG to train a policy which is pre-traine pre-trained in supervised way. Kendall et al. [30] train their deep RL algorithm onboard to efficiently learn to drive a real-world vehicle. The perception task is decoupled out of the online RL process in [50, 9, 62]. The model- method WoR [12) assumes world on rails and uses policy distillation to realize powerful performance.
Imitation learning (IL), especially behavior cloning, collects recorded data for models to mimic with high data efficiency. The expert data typically has two forms, trajectories and control actions. Zeng et al. [58] train a cost volume to generate the planning route, while [49, 8, 25] explicitly design safety and comfort costs based on semantic occupancy maps to select the best one in the expert trajectory sets. Zhang al. [59] predict trajectories of surrounding vehicles with labeled BEV map. LBC [1i] and NEAT [15] decode waypoints from a dense heatmap or offset map. These approaches aforementioned all utilize relatively dense representation obtain results which increases model complexity. Transfuser and its variants [46, 29] adopt a simple GRU to auto-regress waypoints. LAV [10) adopts temporal GRU module to further refine the trajectory. They unanimously achieve impressive performance on the CARLA leaderboard, motivating us to adapt the auto-regression scheme as well in our design. On the other hand, all trajectory- get the ultimate actions, which may cause inferior effects in complicated scenarios.
methods use PID controllers to
Another genre to predict control actions directly is proposed in [44, 40, 54J. CIL [17] adds measurement encoder and multiple branches for different high level commands with the image encoder. CILRS [18] is proposed afterwards and further introduces a speed prediction head. They stand as classic baselines for IL in the CARLA driving simulator. Diverse optimized approaches presented based on them, such as multi-modal inputs [22, 53], multi-task learning [56, 37, 24, 27, 31, 26, 63], dataset aggregation [45] and knowledge distillation [61, 60). However, the compact control-based methods often have higher vehicles collision rates, remaining an interesting domain to explore. Similar work exists in other related domains such as robotic navigation as well. [43] learns a controller after local trajectory planner improve the overall navigation behavior.
2.2 Multi-task and Ensemble Learning for Autonomous Driving
Multi-task learning a popular approach to train several related tasks simultaneously to help each other and improve generalization [7, 2]. Combinations of various autonomous driving tasks such
3

---

as object detection, lane detection, semantic segmentation, depth estimation, etc. have been proved
to be capable of achieving incredible performance [38, 14, 47, 34, 52, 13]. MTL is also suitable in the end-to-end problem since it is observed the performance of a direct mapping from an image to control signals is limited. [56] adds a speed prediction task similar to CII. [17] and [63] separates the lateral and longitudinal controls as two tasks. LAV [10] trains an extra scene mapping network, and [24, 27, 29] additionally predict optical flow or dense depth. Our idea of training trajectory and control simultaneously is closely related to FASNet [31]. FASNet predicts future positions of the ego agent as an auxiliary task and adds a kinematic loss considering the relation between control and locations. However, the constrain is based on constant velocity model which neglects the important throttle and brake, and it does not work at the inference time. On the other hand, our TCP framework has feature interactions at an earlier stage to fully explore their potential mutual benefits.
Ensembles of models have long been utilized to improve the performance in computer vision [19, 33, 35, 51, 55, 45]. Besides the normal combination of models, two classic ensemble learning methods are particularly preferred in the autonomous driving regime. One the Test-Time Augmentation (TTA), which is of great help to the 3D object detection task with LiDAR [6, 36]. Another one is the fusion of experts [28] where experts are trained on a subset of the input space and gating network is trained to provide the fusion weights. LSD [42] and MoDE [32] divide dataset into sub-scenarios to get different sub-policies for end-to-end autonomous driving. These traditional ensemble approaches combine models of the same structure while our approach tries to combine two different representations. Also, the multiple experts design increases the complexity of the training strategy and we seek to have simpler situation based fusion scheme to boost the performance.
3 Trajectory-guided Control Prediction
3.1 Problem Setting
Problem formulation. Given the state x comprised of the sensor signal i, the speed of the vehicle and the high level navigation information g including a discrete navigation command and the coordinates of navigation target provided by the global planner, the end-to-end model needs to output control signals comprised of longitudinal control signals throttle [0. 1] and brake [0, 1]. and
the lateral control signal steer E [-1,
Conventional methods tackle this problem with either trajectory-output or control-output only model. However, TCP combines both of them as two branches: a trajectory branch which predicts the planned trajectory and a control branch which is guided by the trajectory one and outputs both current and multi-step control signals into the future. Both branches are trained a supervised manner. Consider an expert which directly outputs the control signals at each step, supervising the predicted trajectory with the ground truth trajectory makes it not strictly satisfy the setting of behavior cloning in imitation learning. The ground truth trajectory indeed involves future expert actions and future states about the environment, so we formulate it as a trajectory planning task with ground truth trajectory supervision for our trajectory branch. As for the control branch, training control model which makes current control prediction supervised by the expert control is just behavior cloning in imitation learning, and it can be formulated as:
arg min Dix(aa") ([(((aam)(())))) (x)))) 8
where D = {(x, a²}} is a dataset comprised of state-action pairs collected from the expert. mp denotes the policy of the control branch, and L is the loss measuring how close the action from the expert and the action from our model is. The expert collects the dataset by controlling the vehicle and interacting with the world. Each collected route is trajectory& & {= (xo, aj. X1, af. (xT) as sequence of state action pairs {(x,, a, o which is then added into the whole dataset D.
Expert demonstration. Here we choose Roach [60] the expert. Roach is simple model trained by RL with privileged information, including roads, lanes, routes, vehicles, pedestrians, traffic lights, and stops, all being rendered into 2D BEV image. Such a learning- based expert can transfer more information besides the direct supervision signals compared with an expert made by hand-crafted rules. Specifically, we have feature loss which forces the latent features before the final output head from the student model to be similar to that of the expert. value loss is also added as an auxiliary task for the student model to predict an expected return.

---

traj
Image
WP, wPk
Trajectory Branch
traj
Guldance
Messurement Enender
Siuution
speed: Turn Left" "
Multi-step Control Branch
a
goal: (xay
ag. a
Lett
Figure 2: Overview of Trajectory-guide Control Prediction (TCP), The encoded features are shared by the trajectory and multi-step control branch. The trajectory branch provides per-step guidance for multi-step control prediction. Outputs from two branches are combined according to our situation based fusion scheme to generate the ultimate control actions.
3.2 Architecture Design
Overview, As illustrated in Fig. 2, the whole architecture is comprised an input encoding stage and two subsequent branches. The input image i goes through CNN based image encoder, such as ResNet [23], to generate a feature map F. In the meantime, the navigation information g is concatenated with the current speed form the measurement input m, then ML. based measurement encoder takes m as its input and outputs the measurement feature jm- The encoded features are then shared by two branches for subsequent trajectory and control predictions. Specifically, the control branch is a novel multi-step prediction design with guidance from the trajectory one, which will be illustrated in detail in the following sections. Finally, situation based fusion scheme is adopted to combine the best of the two output paradigms, We will go over each part in detail below.
3.2.1 Trajectory planning branch
Different from control prediction which directly predicts control actions, the trajectory planning branch first generates a planned trajectory comprised of waypoints at K steps for the agent to follow, and then the trajectory is processed by low-level controllers to get the final control actions. With the shared feature from the input encoder, the image feature map F is average pooled and concatenated with the measurement feature jm to form juraj, Inspired by [46], we feed jtraj into GRU [16] to auto-regressively obtain future waypoints one by to form the planned trajectory altogether.
We have two PID controllers for longitudinal and lateral control respectively. With the planned trajectory, we first calculate the vectors between consecutive waypoints. The magnitudes of these vectors represent the desired speed and are sent to the longitudinal controller to generate throttle and brake control actions, and the orientations are sent to the lateral controller to get the steer action.
3.2.2 Multi-step control prediction branch
As discussed in Sec. 3. for control model predicting current control actions based on current input only, the supervised training is just behavior cloning, which relies on the independent and identically distributed (IID assumption. This assumption apparently does not hold because the distribution shifts in test cases, since the closed-loop tests require sequential decision making where the historical actions will affect the future states and actions. Instead modeling it as a Markov Decision Process (MDP) and resorting to reinforcement learning, here we devise simple way to mitigate the problem by predicting multi-step control into the future.
Given the current state Xt, now our multi- step control prediction branch outputs multiple actions:
T9 Tupuitt = (a., a1+ ... a at+k). However, is difficult to predict future control actions since we only have sensor inputs at the current time step. Towards this problem, we devise temporal module to implicitly earry out the changing and interaction process of the environment and our agent. supposed to provide mainly dynamic information about the environment and the status of the agent
5

---

itself, such as the motion of other objects, the changing of traffic lights, and the status of the ego agent. Meanwhile, improve the ability of incorporating critical static information (e.g., curbs and lanes) and boost the spatial consistency of two branches, we propose to use the trajectory branch to guide the control counterpart to attend to proper regions of the input image at each future time step.
Temporal module. Our temporal module is implemented with GRU for better consistency with the trajectory branch. At step (0 S K 1). the input for the temporal module is the concatenation of the current feature jr (more construction details in the next section) and current predicted action a, which is a compact representation about the current states of the environment and the agent itself. The temporal module is supposed to reason about the dynamic changing process based current feature vector and the predicted action. Then the updated hidden state h will contain dynamic information about the environment and the updated status of the agent at time step t. + 1. To some extent, the temporal module acts as a coarse simulator with the whole environment and the agent being abstracted as a feature vector. It then simulates the interaction between the environment and the agent based on current prediction of actions.
Trajectory-guide attention. With the sensor input at current step only, it is hard to pick out desirable regions where the model should focus on at future steps. However, the location of the ego agent contains important cues about how to find those regions containing critical static information for control prediction at each step.
Therefore, we seek help from the trajectory planning branch to get information about the possible location of our agent at that corre- sponding step. As shown in Fig. 3, TCP im- attention map
WPt
traj
Waypoint GRU
plements this by learning to extract important information from the en- coded feature map. The interaction between two branches enhances the consistency of these two strongly related output paradigms and further elaborates the multi-task spirit. Specifically, with the 2D feature map tracted by the image encoder F at time step t S t < K), we calculate an attention map W. E R kx using the correspond- ing hidden states from the control branch and the trajectory branch:
F
Temporal Module
Wr
Figure 3: Detailed trajectory guiding process. For predictions time step the hidden from the waypoint GRU the temporal module are combined learn an attention weight map to re-aggregate the 2D image feature map for control prediction.
W. MLP(Connca(lh traj I,"))..
The attention map w. E I is adopted to aggregate the feature map F for this step. We then combine the attended feature map with heth to form the informative representation feature jet containing both static and dynamic information about the environment and the ego agent. The process can be described as follows:
jet
CoaCooonattSmmSStttttttnaa (w:) F), 1).
(3)
The informative representation feature jett is fed into policy head which is shared among all time steps to predict the corresponding control action a, Note that for the initial step, we only use the measurement feature to calculate the initial attention map and combine the attended image feature with the mcasurement feature to form the initial feature vector jo To guarantee the feature jrs does describe the state at that step and contain the important information for control prediction, we add feature loss at each step to make just close to the feature of the expert as well.
To this end, our TCP framework endows the model with the reasoning ability along short time horizon. It emphasizes how to make current control prediction close the one from the expert. Furthermore, it takes into account what current control prediction can make the environment states and status ego agent in future time steps similar to the ones from the expert.
3.3 Loss Design
Our loss contains trajectory planning loss Liraj, control prediction Lott, and auxiliary loss Cowz.

---

For the trajectory planning branch, the loss Liraj can be expressed as:
Liraj
wP: wp, + dp Lp
(straj EEpertt (Jo +J0
where wPt, wPt are the predicted and ground truth waypoint at the 1 step respectively. Lp indicates the feature loss measuring the L2 distance between Jo and the feature Jo EExpert from the expert at the current step additional supervision signal [60]. Xf is tunable loss weight.
For the control prediction branch, we model the action as beta distribution. The loss Lett is:
K
Leu KL (Beta(ap)) (etaa(())) +
KL (Bta((a))
K
t=1
(5)
K
+ Ap LF (J0t
Export
Σ
LF (JEt) .Je J.apport)
K
where Beta(a) denotes the beta distribution represented by the corresponding predicted distribution parameters and KL-divergence is used measure the similarity between the predicted control distribution and the one from expert, i.e., Beta(ā). Feature loss is applied here as well. Note that all losses for future time steps (t 2 1) are averaged and then added the loss for the current time step (t = 0), since the action executed immediately should be our key target to optimize.
To help the agent better estimate its current state, add speed prediction head to predict current speed from the image feature and value prediction head to predict the expected return estimated by the expert, similarly in [60]. We take the L, loss for the speed prediction and T-2 loss for the value prediction, denoting their weighted sum as Laxa-
The overall loss is as follows, weighted by Àeraji dest Aauxi
= Aeraj Liraj + deel" Loll + daux Louz-
(6)
3.4
Output Fusion
Algorithm 1: Situation based fusion scheme to com-
We have two forms of output representations from our TCP framework: the planned tra- jectory and the predicted control. To fur- ther combine their advantages, we devise a situation- slase fusion strategy depicted in Algorithm 1. Specifically, denote a as a com- bination weight whose value is between 0 to 0.5, in a certain situation where one represen- prior
bine the two output paradigms
Input: sensory input speed of the ego vehicle v, high level navigation information g. Hyper parameters :ombination weight a E [0, 0.5)
Output: final control signals a
(wp,)) Jtzo, at +- TCP(i, v, g) a traj Low-level Controller ((wP. Get current situation situation is trajectory specialized then
tation is more suitable according belief, we combine the results from trajectory and control predictions by taking average with weight a so that the more suitable one takes
(1 a) aa" traj
else end
a x
+ (1 - a) x at)
up more weight -a))). a). Note that the com- bination weight indeed does not need to be constant or symmetric, which means we can set different values under different situations or different for specific control signals. In our experiment, we choose the situation according to whether the ego vehicle is turning, implying that if is turning, the situation is control specialized otherwise trajectory specialized.
4 Experiments
4.1 Experimental Setup
Task & Evaluation metrics. Our method is validated and tested in the CARLA driving simulator [20]. Given route defined by a sequence sparse navigation points together with high level commands (straight, turn left/right, lane changing, and lane following), the closed-loop driving task
7

---

Table 1: Evaluation on the public CARLA Leaderboard [1] (accessed in May 2022). Our method TCP and TCP-Ens achieve a driving score of 69.714 and 75.137 respectively with only monocular camera, More detailed infraction statistics can be found in the Supplementary.
Sensor Inputs
Key Metrics
Rank
Method
Driving Score
Route Completion
Infraction Score
#Cameras
LIDAR
TCP-Ens (ours)
75.137
85.629
0.873
1
TCP (ours)
X
69.714
82.962
0.851
TCP-SB (ours)
X
68.695
82.957
0,833
LAV [10]
61.846
94,459
0.640
3
Transfuser
61.181
86.694
0.714
4
Latent Transfuser
45.029
75.366
0.618
5
GRIAD 19]
36.787
61.855
0.597
Transfuser+ [29]
34.577
69.841
0,562
WoR [12]
31.370
57.647
0.557
MaRLn [50]
24.980
46,968
0.518
9
NEAT [15]
21.832
41.707
0.650
Table 2: Comparison between the control and trajectory only model in terms of infractions frequency. TurnRatio means the corresponding ratio of happening during turning.
Driving Score
Collisions vchicles
Collisions layout
Off-road infractions
Agent blocked
Model
#/km
TumRatio
#/m
TurnRatio
#/km
TurnRati
#kmm
TurnRatio
Control-Only
32.45.12.23
1.25 0.85
50.90%
0.23
10.00%
0.59
46.15% 62.90%
0.41 0.77
50.00%
Trajectory-Only
28.29:13.03
38.70%
0.77
64.20%
0.74
64.20%
requires the autonomous agent drive towards the destination point. is designed to simulate realistic traffic situations and includes different challenging scenarios such as obstacle avoidance, crossing an unsignalized intersection, and sudden control loss. There are three major metrics: Driving Score, Route Completion, and Infraction Score. Route Completion is the percentage of the route completed by the autonomous agent. Infraction Score measures the number of infractions made along the route, with pedestrians, vehicles, road layouts, red lights, and etc. Driving Score is the main metric which is the product of Route Completion and Infraction Score.
We use randomly generated routes under random weather conditions to collect 420K data in the public towns offered by the CARLA simulator. Similar to [10], we train TCP on 189K of data in 4 out of 8 towns (Town01, Town03, Town04, and Town06) for ablations and train with all 420K data for our online leaderboard submission.
Dataset.
4.2 State-of-the-art Comparison
Table shows the result of the comparison between method and the top entries on the public CARLA Leaderboard [1]. We report the results of TCP and two variants. TCP-SB replaces shared encoders of TCP with two separate ones for two branches, and TCP-Ens the ensemble of TCP and TCP-SB. Our method TCP-Ens ranks first on the Icaderboard with 75.137 driving score and highest infraction score, and TCP alone also surpasses prior methods. Note that our method only uses monocular camera while the top 2-4 methods all use multiple cameras and a LiDAR. Our driving score 50.157 higher than the second-best monocular camera method, MaRLn [50]. Our route completion is slightly inferior to the LiDAR candidates one reason is that methods using LiDAR may have a better object detection ability. Based on the detection results, they usually adopt a crawling strategy, indicating that the vehicle would move slowly when it has stopped for long time and there are no obstacles ahead. As described in [29], this could alleviate ego vehicle's 's blocking problems to boost the route completion performance.

---

The drfectory-guideer attention maps in two cases. In each case (row), from the left to right we show that the input image with the predicted trajectory (the first waypoint is projected out of the image), the predicted trajectory in the top-down view, the attention map W1, the attention map W3-
Figure 4:
Table 3:
Ablative study on the effectiveness of different components design of our model.
Table 4: Comparison between MTL and semble methods (a is 0.3 for all experiments).
Driving Score
Route Completion
Infraction Score
Driving Score
Exp.
Exp.
#Param.
FLOPs
FPS
Control
32.45:+22.23
76.54+3.22
0..0
Ensemble
45.03 1.28
46.81M
17.07G
69.47
+ traj-task
14.98:11.96
5,,32:5.50
0.49:10.05
MTL
48.27::0.58
23.58M
8.54G
133.30
+ temporal
42.87+44.77
87.51=3.63
0.49+00.07
TCP-SB
52.46+44.66
47.26M
17.07G
69.35
traj-attn
46.08+3.44
84.95-1.84
0.56++0.03
TCP
57.01+1.88
25.77M
8.54G
125.71
+ fusion
5..01+1.888
85.27+1.20
0.67+00.01
TCP-Ens
59.09+3.66
73.03M
25.61G
44.70
4.3 Control vs. Trajectory
In this section, we conduct quantitative experiments to compare the Control-Only model and the trajectory-Onle model to demonstrate their advantages and disadvantages. For both models, we use the same setting except for the output head and its corresponding loss. We use ResNet-34 encode visual inputs and a measurement module to encode the navigation information. Similar to [60], we add speed and value heads as auxiliary tasks to help the model better encode the environment. For Control-Only,
predict the control distribution based on the concatenated latent feature from the two encoders. As for Trajectory-Oni we feed the feature to a GRU decoder to generate waypoints. As shown in Table though Trajectory-Onif collides with vehicles less frequently than Control- Onlyy, it has more layout collisions, oof-road infractions, and agent blocks. We also count the ratio of each kind of infraction that occurs during turning. It can be observed that for Trajectory-Oniy portion of such infractions happen when the ego agent is turning compared to Control-Only This has verified that Prajectory-Only performs worse when the agent turning, which is probably caused by the unsatisfactory trajectory following performance of simple PID controllers as discussed in Sec. i. As for the fact that Control-Only has higher vehicle-collision rate, is because the model focuses on the current time step and the reaction to potential collisions tends to be late, as depicted in Sec. 1 as well. The results above further validate the necessity of combining the two output paradigms.
large
4.4 Ablative Study and Visualization
Component analysis. We first validate the effectiveness of the trajectory-guide guided multi-step control prediction design, as shown in Table 3. We only employ the control branch output except for the last complete one when fusion applied for these ablations. Adding a trajectory branch as an auxiliary task improves the performance by 2.5 points. The multi-step predictions with our temporal module greatly help with 7.9 points gain, and adding the trajectory-guided attention further acquires an improvement of 3.2 points. Finally, applying our situation based fusion scheme (a is set to 0.3) significantly boosts the infraction score, leading the overall driving score to 57.
Multi-fask vs. Ensemble. The comparison regarding their performances and computational com- plexity given in Table 4. Ensemble denotes directly combining the outputs of Control-Only and Trajectory-Oni with our situation based fusion scheme. MTL represents the model with shared CNN backbone and measurement encoders followed by a trajectory branch and a control branch, but the control branch predicts current step prediction only and there are no interactions between the two branches. We conclude that directly combining two models with our fusion scheme greatly improves the performance, and using an MTL approach works better than ensemble but with much

---

smaller model size and GFLOPs. A conventional ensemble approach to combine results from TCP and TCP-SB as TCP-Ens brings further performance gain the cost of computational complexity.
Situation based fusion weight. We investigate the choice of the combination weight in the situation based fusion scheme and show the box plot of the driving scores in the fig- the right. Besides a E [0, 0.5]. we additionally test 0.7 and 1, meaning that two results are conversely mixed with our specialization definition. We see that only using the control from the specialized branch a = 0 performs poorly while directly taking the average or fusing conversely still has comparable results. One reason is that the situation criterion used here is whether the vehicle turning, making most cases trajectory specialized, and the stronger con- trol branch is not utilized enough if a is small. Note that the situation based fusion scheme is general and flexible, and the criterion or a value used here is relatively coarse.
Noho
Figure 5: Box plot the driving score with different a values (3 trials for each a)).
Visualization. Fig. 4 visualizes the trajectory-guided attention maps. The trajectory branch provides location-relace related information guide the control branch focus on important regions which are useful for future control prediction. See more qualitative results in the Supplementary.
Conclusion
In this work, study two learning and prediction paradigms based on trajectory and direct control, respectively. for end-to-end autonomous driving. We propose unified framework comprised of trajectory branch and a novel multi-step control branch with interactions between. We design situation based fusion scheme to combine the results from two branches. Our method with only monocular camera has achieved state-of-the-a performance on the CARLA Leaderboard.
Acknowledgments
This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), NSFC (62206172, 61972250). Shanghai Municipal Science and Technol- ogy Major Project (2021SHZDZX0102) and Shanghai Committee of Science and Technology (21DZ1 100100, 22511105100).
References
CARLA autonomous driving lcaderboard. https://lesderboard waw.aaaaaar....aaa.aa.aaaaaaaaa. org/.2022.
2022. 3,
[2] Andreas Argyriou, Thcodoros Evgeniou, and Massimiliano Pontil. Multi-task fcature learning. In NeurlPS, 2006. 2, 3
[3] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.07316, 2016.
[4] Mariusz Bojarski, Chenyi Chen, Joyjit Daw, Alperen Degirmenci, Joya Deri, Bemhard Firner, Beat Flepp. Sachin Gogri, Jesse Hong, Lawrence Jackel, et al. The nvidia pilomett experiments. arXiv preprint 08877 2020.
[5] Eduardo Camacho and Carlos Bordons Alba. Model predictive control. Springer science business media, 2013. 2
Manuel Carranza-Garefa, Pedro Kara-Benitez, Jorge Karcfa-Gutiérrez and José C Riquelme. Enhancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance. Neurocomputing, 2021. 4
Rich Caruana. Multitask lcarning. Machine learning, 1997. 2,
[8] Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unitied model to map, perceive, predict and plan. In CVPR, 2021. 2,
10

---

[9] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde. Gri: General reinforced imitation and its application to vision-based autonomous driving. arXiv preprint arXiv22111.0575. 2021.
1, 3,
[10] Dian Chen and Philipp Krähenbühl. Learning from vehicles. In CVPR, 2022. 1,3,4,
[11] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl, Learning by cheating. In CoRL, 2020.
2, 3
[12] Dian Chen, Vladlen Koltun, and Philipp Krähenbühhl, Learning to drive from a world on rails. In ICCV,
2021. 3, 8
[13] Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. arXiv preprint arXiv:2203.11089) 2022. 2,
[14] Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi task learning. CVPRW, 2019.
[15] Kashyap Chitta, Aditya Prakash, and Andreas Geiger. Neat: Neural attention fields for end-to-end autonomous driving. In ICCV, 2021. 1, 2, 3,8
[16] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. 2,5
[17] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In ICRA, 2018. 1, 3,44
[18] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. ICCV, 2019. 1.2.3
[19] Thomas G Dietterich. Ensemble methods in machine learning. MCS. 2000.
[20] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, 2017. 3,
[21] Hongyan Guo, Dongpu Cao, Hong Chen, Zhenping Sun, and Yunfeng Hu. Model predictive path following control for autonomous cars considering a measurable disturbance: Implementation, testing, and verification. MSSP, 2019.
[22] Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Prze- mystaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, al. Urban driving with conditional imitation learning. ICRA, 2020.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 5
[24] Yuenan Hou, Zheng Ma. Chunxiao Liu, and Chen Change Loy. Learning to steer by mimicking features from heterogeneous auxiliary networks. In AAAI, 2019. 3,4
[25] Shengehao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end rision-bascd autonomous driving via spatial-temporal feature learning. In ECCV, 2022, 2,
[26] Sehastian Huch, Aybike Ongel, Johannes Betz, and Markus Lienkamp. Multi-task end-to-end self-driving architecture for cav platoons. Sensors, 2021.
[27] Keishi Ishihara, Anssi Kanervisto, Jun Miura, and Ville Hautamaki. Multi-task learning with atlention for end-to-end autonomous driving. CVPR, 2021. 3,4
[28] Rohert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 1991. 4
[29] Bernhard Jaeger. Expert drivers for autonomous driving. Master's thesis, University of Tübingen, 2021. 1,
2, 3, 4,
[30] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Diew Lam, Alex Bewley, and Amar Shah. Learning to drive a day. In ICRA, 2019.

---

[31] Inhan Kim, Hyemin Lee, Joonyeong Lee. Eunseop Lee, and Daijin Kim. Multi-task learning with future states for vision-based autonomous driving. In ACCV, 2020. 3,
[32] Inhan Kim, Joonycong Lee, and Daijin Kim. Learning mixture of domain experts via disentangled factors for autonomous driving. AAAI, 2022.
[33] Philipp Krähenbühl and Vladien Koltun. Learning to propose objects. In CVPR, 2015. 4
[34] Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sitsu, Christian Witt, Isabelle Leang, Stefan Milz, and Patrick Mäder. Omnidet: Surround view cameras based multi-tast task visual perception network for autonomous driving. RA-L, 2021. 2,4
[35] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NewrlPS, 2017. 4
[36] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo Wu, Yifeng Lu, Denny Zhou, et al. Deepfusion: Lidar-camera deep fusion for multi-modas 3d object detection. In CVPR, 2022. 4
[37] Zhihao Li. Toshiyuki Motoyoshi, Kazuma Sasaki, Tetsuya Ogata, and Shigeki Sugano. Rethinking self- driving: Multi-task knowledge for better generalization and accident explanation ability. arXiv preprint arXiv: / 1000 2018. 3
[38] Ming Liang, Bin Yang. Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. In CVPR, 2019. 2,4
[39] Xiaodan Liang, Tairui Wang, Luona Yang. and Erie Xing. Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In ECCV, 2018. 1,3
[40] Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann Cun. Off-road obstacle avoidance through end-to-end learning. NewrIPS, 2005.
[41] Urs Muller. Jan Ben, Eric Cosatto, Beat Flepp, and Yann Cun. Off-road obstacle avoidance through end-to-end learning. In NeurlPS, 2005. 2
[42] Eshed Ohn-Bat Aditya Prakash, Aseem Behl, Kashyap Chitta, and Andreas Geiger. Learning situational driving. In CVPR, 2020. 1.4
[43] Ashwini Pokle, Roberto Martín-Marttn Patrick Goebel, Vincent Chow, Hans M Ewald, Junwei Yang. Zhenkai Wang, Amir Sadeghian, Dorsa Sadigh, Silvio Savarese, al. Deep local trajectory replanning and control for robot navigation. In ICRA, 2019. 3
[44] Dean A Pomerleau. Alvinm: An autonomous land vehicle neural network. In NewrlPS, 1988. 2,
[45] Aditya Prakash, Aseem Behl, Eshed Ohn-Bar, Kashyap Chitta, and Andreas Geiger. Exploring data aggregation policy learning for vision-based urban autonomous driving. In CVPR, 2020. 3, 4
[46] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modas modal fusion transformer for end-to-end autonomous driving. In CVPR, 2021. 3.5.
[47] Georgia Rajamanoharan, Aytaç Kanacı, Minxian Li, Shaogang Gong, et al. Multi-task mutual learning for
vehicle re-identification. In CVPR, 2019, 4
[48] Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference. planning, and control. In ICLR, 2019.
[49] Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In ECCV, 2020.
[50] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. CVPR, 2020. 3,
[51] Apoorv Vyas, Nataraj Jammalamadaka. Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore Willke. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In ECCV, pages 550-564 2018. 4
[52] Dong Wu, Manwen Liao, Weitian Zhang, and Xinggang Wang. Yolop: You only look once for panoptic driving perception. arXiv preprint arXiv:2188.11250 2021, 4
12

---

[53] Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu. and Antonio M López. Multimodal end-to-end autonomous driving. T-ITS, 2020. 3
[54] Huazhe Xu. Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. CVPR, 2017.
[55] Jie Xu, Wei Wang. Hanyuan Wang, and Jinhong Guo. Multi-model ensemble with rich spatial information object detection. Pattem Recognition, 2020. 4
[56] Zhengyuan Yang, Yixuan Zhang, Jerry Yu, Junjie Cai, and Jiebo Luio. End-to-end multi-modal multi-task vehicle control self-driving cars with visual perceptions. ICPR, 2018. 3,4
[57] Éloi Zablocki, Hédi Ben- Ben-Younes, Patrick Pérez, and Matthicu Cord. Explainability of vision-based autonomous driving systems: Review and challenges. arXiv preprint arXiv:2101.05307, 2021. 2
[58] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
[59] Jimuyang Zhang and Eshed Ohr-Bar. Learning by watching. In CVPR, 2021. 1,3
[60] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating reinforcement learning coach. In ICCV, 2021. 1, 3.4.441 7.9
[61] Albert Zhao, Tong Hc, Yitao Liang, Haibin Huang, Guy Van den Broeck, and Stefano Soatto. Sam: Squeeze-and-mimic networks for conditional visual driving policy learning. In CoRL, 2021.
[62] Yinuo Zhao, Kun Wu, Zhiyuan Xu, Zhengping Che, Qi Lu, Jian Tang, and Chi Harold Liu. Cadre: cascade deep reinforcement learning framework rision-based autonomous urban driving. AAAI, 2022. 3
[63] Zeyu Zhu Huijing Zhao. Multi-task conditional imitation learning for autonomous navigation at crowded intersections. arXiv preprint 2xX1v:2202.101244 2022. 3,4
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section XX.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Supplementary.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the Supplementary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
13

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] The zip file of source code and the URL of data are in the Supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1 and the Supplementary.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Sec. 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the Supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use the CARLA
simulator and pretrained ResNet model.
(b) Did you mention the license of the assets? [Yes] See the Supplementary.
(c) Did you include any assets either in the supplemental material
URL? [Yes] The zip file of source code and the URL of data are in the Supplementary.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] CARLA is public simulator.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] Data in the public simulator does not have these issues.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rTkwPauIJq", "reviewer": "Reviewer_ZcJ9", "summary": "This paper presents TCP, a novel trajectory predicting/controlling network architecture for vision-based end-to-end autonomous driving. At test time, TCP ensembles the control outputs from a direct prediction branch and a trajectory prediction  w/ PID controllers. Both branches use GRU to predict waypoints/controls auto-regressively and are jointly trained. At each timestep, the control prediction branch additionally takes as input attention-weighted image backbone features computed from the trajectory prediction branch.\n", "questions": "Apart from the ablation mentioned in the Strengths & Weaknesses section, I would like to hear from the authors how they think TCP will work in a real world setting, what might be the main obstacles and bottlenecks that the proposed method could have when scaling up to real world data.\n", "limitations": "Same as my question in the section above – I would like to hear from the authors what they think might be the limitations of the proposed method when scaling to the real world.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n+ The strongest aspect of this paper is its impressive performance on the challenging CARLA leaderboard. The fact that TCP is capable of attaining such a strong performance with a simple vision backbone is inspiring, suggesting prior methods for end-to-end driving in CARLA might all have suboptimal controller designs. \n+ The ablation studies are mostly comprehensive, and the design choices are well justified. \n+ The attention map visualization is insightful and shows what areas are attended for the control prediction task.\n\n=== Weaknesses ===\n- I do not have any major complaints regarding the technical content. My main question is how much performance gain comes from fusing the trajectory/control prediction branches as the title and paper implies, or suboptimal PID parameters for the trajectory-only branch, and the trajectory-attended image features inputs in the autoregressive prediction stage. I would like to see a trajectory-only variant where, similar to the proposed situation-based fusion, has situation-based PID controller parameters (different PID gains based on different high-level commands, or whether it’s turning etc.), which still has the novel trajectory-computed attention fusion module (i.e. the attention-weighed image features get consumed by the trajectory branch instead of the control branch). This will further help distinguish the main source of improvements among the design choices.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eS4cC7EOJh8", "reviewer": "Reviewer_cv3D", "summary": "This work presents a novel approach (TCP) that combines trajectory planning and control prediction in a multitask learning framework for end-to-end autonomous driving. It analyzes the limitations of both paradigms (Fig. 1), eg. inertial problem, incorrect turnings, and single-step prediction, and proposes a multi-step control prediction module with trajectory-guided attention and situation-based fusion scheme to incorporate the best of both worlds. Extensive experiments on the CARLA leaderboard (Table 1) show state-of-the-art performance by a wide margin, with only a monocular camera, compared to baselines that use multiple sensors as input. Moreover, the ablation study (Table 2,3,4) and visualizations (Fig. 4 and supplementary) provide valuable insights into the capabilities of the proposed approach.", "questions": "Based on the weaknesses mentioned above, the following experiments are required to verify the claims in the paper:\n- Trajectory-only model with feature loss applied to each prediction timestep and auxiliary loss (on speed and value prediction) used as well. This requires training a new model and should be compared in Table 3.\n- A single GRU module to predict both future waypoints and multi-timestep control predictions from the hidden state of the GRU. This requires training a new model and should be compared in Table 3.\n- Direct ensemble (average control predictions) of control-only and trajectory-only models in Table 4. This does not require training any model.\n- An ensemble of multi-step control prediction and trajectory prediction (both situation-based fusion and direct averaging should be considered) without the trajectory-guided attention module. This does not require training any model and should be compared in Table 4.\n\nCertain parts need clarification:\n- L194-195: How does the multi-step control prediction branch mitigate the IID assumption?\n- L202-204: How does the trajectory branch improve the ability to incorporate static layout information (eg. curbs)?\n- L309-L313: Why does TCP has lower route completion compared to LiDAR-based methods. Is it because the TCP gets blocked more often than other methods? It is also possible that taking the maximum of brake values from different models for the ensemble (L33-34 in supplementary) leads to an overly conservative policy.\n- The evaluation protocol for the experiments in Table 2,3,4 is different than Table 1 which uses the CARLA leaderboard. The authors should include a brief description of this evaluation protocol in the paper.\n - Fig. 1(a) shows a failure case at turnings for trajectory-based methods. Since the details of the PID controllers are not provided in the paper, it is unclear if the PID controller could just be tuned to better follow the trajectory.\n - Implementation details regarding the GRU modules, MLPs in the trajectory-guided attention, and weights for loss terms (Eq. 4,5,6) should also be included in the paper.\n\nAdditional suggestions to improve the paper:\n- As mentioned earlier, it would be helpful if the authors could also provide comparisons to FASNet. This would greatly improve the paper.\n- It would be interesting to visualize the GradCAM (Selvaraju et al. IJCV 2019) attention maps for the multi-step control prediction branch without the trajectory-guided attention module. This could provide useful insights into whether the multi-step control prediction helps the model to focus on important regions for future control predictions. These GradCAM attention maps can be compared to attention maps in Fig. 4 to verify the utility of trajectory-guided attention.", "limitations": "The authors have provided a discussion on limitations and societal impact in the supplementary. Additional suggestions are mentioned above.  \n\n### Update after Rebuttal ###\nI appreciate additional ablations and discussions with authors which helped me get a better understanding of the paper. After rebuttal and discussion with other reviewers, I am retaining my original score. \n\nMy main concern is that majority of the gain is coming from the heuristic $\\alpha$ used for ensembling. Table 2 in the main paper shows Control-only=`32.45`, trajectory-only=`28.29`. In one of the ablations provided by the authors in the rebuttal, ensemble of control-only & trajectory-only methods with non-constant $\\alpha$ gives a score of `50.98`. TCP has a score of `57.01` in Table 4 in the main paper. The ensemble heuristic leads to a gain of around `19` points whereas all the architectural modifications - shared backbone, multi-step control, temporal module, trajectory-guided attention - combined result in a gain of `6` points. \n\nWhile it might be possible to use the fusion mechanism in a general and flexible way (eg. as a mixture of experts or using probabilistic uncertainty, as mentioned by the authors in the rebuttal), the paper currently uses a heuristic for fusion. Based on this, I agree with reviewer `6yad` that the paper does not have sufficient technical contribution for NeurIPS, and venues like IROS, ICRA are a better fit. The authors have mentioned multiple interesting directions, eg. link between the temporal module and world models, using mixture of experts or probabilistic uncertainty for fusion. I encourage the authors to explore these directions.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea to combine trajectory planning and control prediction in a multitask learning framework is simple and easy to understand. Algorithm 1 and Fig. 2, 3 are great and provide a clear understanding of the proposed approach.\n- TCP achieves state-of-the-art results on the official CARLA leaderboard, surpassing the previous best model by 13.29 points. This is quite impressive since TCP uses a single monocular camera as input whereas the other top models use multiple sensor inputs: multiview camera and LiDAR.\n\nWeaknesses:\n\n- The loss function for the control branch (Eq. 5) contains feature loss for the entire prediction horizon whereas that is not the case for the trajectory branch (Eq. 4). Since the trajectory branch contains a GRU that outputs a hidden state at each prediction timestep, the feature loss (from t=1 to K) could also be applied in Eq. 4 at each future timestep. The authors should include an ablation with a trajectory-only model with feature loss applied to each prediction timestep and auxiliary loss (on speed and value prediction) used as well.\n\n- What is the reason for using 2 separate GRU branches for predicting waypoints and control values? Since both waypoints and control are alternate representations of driving behavior, they could be directly predicted from a single branch. Since the waypoint branch also contains a GRU, the hidden state of that GRU could be used to predict both waypoints and control values through separate MLP heads. It is not clear why a separate temporal module is required since it is already present in the trajectory branch.\n\n- The authors should compare to a baseline which does a direct ensemble (average control predictions) of control-only and trajectory-only models in Table 4. This is important to understand if the situation-based fusion scheme is indeed better than simple averaging.\n\n- Does the Control-Only model in Table 2 and L340 predict only single-step control values? If yes, the authors should also compare to a baseline that does an ensemble of multi-step control prediction and trajectory prediction (both situation-based fusion and direct averaging should be considered) without the trajectory-guided attention module.\n\n- L121-122 mentions that the proposed idea is closely related to FASNet. It would be helpful if the authors could also provide comparisons to FASNet. Since the code of FASNet is not publicly available, implementing, training and testing may not be possible in time for rebuttal but this would greatly improve the paper.\n\n- Sec. B in supplementary mentions that for online submission to the CARLA leaderboard, $\\alpha=0.5$ is used if situation is trajectory specialized whereas $\\alpha=0$ is used when it is control specialized. Also, the maximum of brake is used instead of taking the average. It'd be great if the authors could explain the reasoning behind using a different fusion scheme. Also, how is $\\alpha$ selected? Is there a separate validation protocol for tuning these values?\n\n- Relevant references that should also be included:\n    - Zhihao Li, Toshiyuki Motoyoshi, Kazuma Sasaki, Tetsuya Ogata, Shigeki Sugano. Rethinking self-driving: Multi-task knowledge for better generalization and accident explanation ability. arXiv 1809.11100, 2018.\n    - Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah. Learning to Drive in a Day. ICRA 2019\n    - Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, Alex Kendall. Urban Driving with Conditional Imitation Learning. ICRA 2020\n    - Nicholas Rhinehart, Rowan McAllister, Sergey Levine. Deep Imitative Models for Flexible Inference, Planning, and Control. ICLR 2020\n    - Albert Zhao, Tong He, Yitao Liang, Haibin Huang, Guy Van den Broeck, Stefano Soatto. SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning. CoRL 2020\n    - Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, Antonio M. López. Multimodal End-to-End Autonomous Driving. arXiv 1906.03199, 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Dpg9GwhGgXy", "reviewer": "Reviewer_6yad", "summary": "The paper studies how to combine image-to-trajectory and image-to-control approaches for autonomous driving in the Carla simulator. Both approaches have their advantages and disadvantages, by combining them the paper is able to achieve a new state of the art on the carla leaderboard only using a front facing camera. To fuse the temporal trajectories and static control predictions, the paper proposes to predict input trajectories which are feed through a temporal model similar to a world model that predicts future features and allows trajectory guided attention in the control branch.\n", "questions": "-Did you investigate DAgger to improve the driving of the proposed agent?\n-Did you evaluate how well the temporal module can predict the future? For example by running the open loop action predictions in Carla or simulating a vehicle model?\n-PID controllers are not well suited for lateral trajectory tracking since the are missing a \"feedforward\" component. However, there are several approaches which can deal with this, such as pure pursuit, curvature based feedforward, or as mentioned MPC. All of them are easy to implement, so I was wondering if using such a controller would change the finding, and result in a different conclusion (mainly trajectory predictions is all you need similar to some works done by Lyft/Woven Planet Level 5).", "limitations": "Both are discussed but only in the supplementary. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well written, clear to follow and achieves impressive results. I especially like the idea of the temporal module in the multi step control branch, which allows to match control inputs and input features. \n\nWeaknesses: My main concern is that I am not sure if this paper fits NeurIPS? There are little technical contributions and interesting links such as the one between the temporal module and world models are not discussed. On the link to world models, the proposed training approach for the temporal module is actually know to not work well, since it is a purely deterministic model which is not able to capture the stochastic future. Potentially a hybrid approach such as RSSM could work better. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2uoCsQDqU-v", "reviewer": "Reviewer_X2A2", "summary": "This paper presents a novel method for autonomous driving based on a single RGB camera on the vehicle. The method combines direct control commands from images into steering, and an indirect method with a policy that outputs local trajectories, which are then converted into steering commands with an analytical PID controller. The feature representation from the images is shared for both the trajectory and the direct control branches. The combination of commands is performed by a hard-coded heuristic method using a weighted mean based on two predefined situations: turning or not turning. The proposed approach provides good results on the CARLA benchmark, even compared to other methods that use more modalities or more RGB images as input.\n", "questions": "- How do you identify the type of driving mode to select a different fusing criterion?\n", "limitations": "- There is not much discussion about limitations, it would be good to have it. Also a clear evaluation of the error cases would be informative. What are the common failures of the solution?\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: the idea of combining a learned method for trajectory prediction and a learned method for motion control is novel in the domain of autonomous car driving. However, it has been explored for other applications in robotics, e.g., Pokle et al. “Deep Local Trajectory Replanning and Control for Robot Navigation”, 2019. I’d recommend extending the search to other application domains and include them in the related work. In any case, the paper brings novel ideas.\n- Significance: the strength of this paper is in the evaluation. The method has been compared to other methods in a public benchmark, CARLA, and provided good results.\n- Clarity: the paper is clearly written and it is easy to follow. There is some confusion with some terms, e.g., the final method is a combination of trajectory based control and direct end-to-end control, but the title seems to indicate something different.\n- Quality: the paper has acceptable quality. The experiments are mainly the CARLA scores. I miss an important experiment: the fusing mechanism. The paper includes ablations using only trajectory-based or direct control, but not an evaluation of the importance of the hardcoded heuristic with the two types of driving cases. Even the information about how to classify in these two types is not clearly provided.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept