Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a novel method for autonomous driving based on a single RGB camera on the vehicle. The method combines direct control commands from images into steering, and an indirect method with a policy that outputs local trajectories, which are then converted into steering commands with an analytical PID controller. The feature representation from the images is shared for both the trajectory and the direct control branches. The combination of commands is performed by a hard-coded heuristic method using a weighted mean based on two predefined situations: turning or not turning. The proposed approach provides good results on the CARLA benchmark, even compared to other methods that use more modalities or more RGB images as input.

strengths_and_weaknesses: - Originality: the idea of combining a learned method for trajectory prediction and a learned method for motion control is novel in the domain of autonomous car driving. However, it has been explored for other applications in robotics, e.g., Pokle et al. “Deep Local Trajectory Replanning and Control for Robot Navigation”, 2019. I’d recommend extending the search to other application domains and include them in the related work. In any case, the paper brings novel ideas.
- Significance: the strength of this paper is in the evaluation. The method has been compared to other methods in a public benchmark, CARLA, and provided good results.
- Clarity: the paper is clearly written and it is easy to follow. There is some confusion with some terms, e.g., the final method is a combination of trajectory based control and direct end-to-end control, but the title seems to indicate something different.
- Quality: the paper has acceptable quality. The experiments are mainly the CARLA scores. I miss an important experiment: the fusing mechanism. The paper includes ablations using only trajectory-based or direct control, but not an evaluation of the importance of the hardcoded heuristic with the two types of driving cases. Even the information about how to classify in these two types is not clearly provided.

questions: - How do you identify the type of driving mode to select a different fusing criterion?

limitations: - There is not much discussion about limitations, it would be good to have it. Also a clear evaluation of the error cases would be informative. What are the common failures of the solution?
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper studies how to combine image-to-trajectory and image-to-control approaches for autonomous driving in the Carla simulator. Both approaches have their advantages and disadvantages, by combining them the paper is able to achieve a new state of the art on the carla leaderboard only using a front facing camera. To fuse the temporal trajectories and static control predictions, the paper proposes to predict input trajectories which are feed through a temporal model similar to a world model that predicts future features and allows trajectory guided attention in the control branch.

strengths_and_weaknesses: Strengths: The paper is well written, clear to follow and achieves impressive results. I especially like the idea of the temporal module in the multi step control branch, which allows to match control inputs and input features. 

Weaknesses: My main concern is that I am not sure if this paper fits NeurIPS? There are little technical contributions and interesting links such as the one between the temporal module and world models are not discussed. On the link to world models, the proposed training approach for the temporal module is actually know to not work well, since it is a purely deterministic model which is not able to capture the stochastic future. Potentially a hybrid approach such as RSSM could work better. 
questions: -Did you investigate DAgger to improve the driving of the proposed agent?
-Did you evaluate how well the temporal module can predict the future? For example by running the open loop action predictions in Carla or simulating a vehicle model?
-PID controllers are not well suited for lateral trajectory tracking since the are missing a "feedforward" component. However, there are several approaches which can deal with this, such as pure pursuit, curvature based feedforward, or as mentioned MPC. All of them are easy to implement, so I was wondering if using such a controller would change the finding, and result in a different conclusion (mainly trajectory predictions is all you need similar to some works done by Lyft/Woven Planet Level 5).
limitations: Both are discussed but only in the supplementary.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work presents a novel approach (TCP) that combines trajectory planning and control prediction in a multitask learning framework for end-to-end autonomous driving. It analyzes the limitations of both paradigms (Fig. 1), eg. inertial problem, incorrect turnings, and single-step prediction, and proposes a multi-step control prediction module with trajectory-guided attention and situation-based fusion scheme to incorporate the best of both worlds. Extensive experiments on the CARLA leaderboard (Table 1) show state-of-the-art performance by a wide margin, with only a monocular camera, compared to baselines that use multiple sensors as input. Moreover, the ablation study (Table 2,3,4) and visualizations (Fig. 4 and supplementary) provide valuable insights into the capabilities of the proposed approach.
strengths_and_weaknesses: Strengths:

- The idea to combine trajectory planning and control prediction in a multitask learning framework is simple and easy to understand. Algorithm 1 and Fig. 2, 3 are great and provide a clear understanding of the proposed approach.
- TCP achieves state-of-the-art results on the official CARLA leaderboard, surpassing the previous best model by 13.29 points. This is quite impressive since TCP uses a single monocular camera as input whereas the other top models use multiple sensor inputs: multiview camera and LiDAR.

Weaknesses:

- The loss function for the control branch (Eq. 5) contains feature loss for the entire prediction horizon whereas that is not the case for the trajectory branch (Eq. 4). Since the trajectory branch contains a GRU that outputs a hidden state at each prediction timestep, the feature loss (from t=1 to K) could also be applied in Eq. 4 at each future timestep. The authors should include an ablation with a trajectory-only model with feature loss applied to each prediction timestep and auxiliary loss (on speed and value prediction) used as well.

- What is the reason for using 2 separate GRU branches for predicting waypoints and control values? Since both waypoints and control are alternate representations of driving behavior, they could be directly predicted from a single branch. Since the waypoint branch also contains a GRU, the hidden state of that GRU could be used to predict both waypoints and control values through separate MLP heads. It is not clear why a separate temporal module is required since it is already present in the trajectory branch.

- The authors should compare to a baseline which does a direct ensemble (average control predictions) of control-only and trajectory-only models in Table 4. This is important to understand if the situation-based fusion scheme is indeed better than simple averaging.

- Does the Control-Only model in Table 2 and L340 predict only single-step control values? If yes, the authors should also compare to a baseline that does an ensemble of multi-step control prediction and trajectory prediction (both situation-based fusion and direct averaging should be considered) without the trajectory-guided attention module.

- L121-122 mentions that the proposed idea is closely related to FASNet. It would be helpful if the authors could also provide comparisons to FASNet. Since the code of FASNet is not publicly available, implementing, training and testing may not be possible in time for rebuttal but this would greatly improve the paper.

- Sec. B in supplementary mentions that for online submission to the CARLA leaderboard, $\alpha=0.5$ is used if situation is trajectory specialized whereas $\alpha=0$ is used when it is control specialized. Also, the maximum of brake is used instead of taking the average. It'd be great if the authors could explain the reasoning behind using a different fusion scheme. Also, how is $\alpha$ selected? Is there a separate validation protocol for tuning these values?

- Relevant references that should also be included:
    - Zhihao Li, Toshiyuki Motoyoshi, Kazuma Sasaki, Tetsuya Ogata, Shigeki Sugano. Rethinking self-driving: Multi-task knowledge for better generalization and accident explanation ability. arXiv 1809.11100, 2018.
    - Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah. Learning to Drive in a Day. ICRA 2019
    - Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, Alex Kendall. Urban Driving with Conditional Imitation Learning. ICRA 2020
    - Nicholas Rhinehart, Rowan McAllister, Sergey Levine. Deep Imitative Models for Flexible Inference, Planning, and Control. ICLR 2020
    - Albert Zhao, Tong He, Yitao Liang, Haibin Huang, Guy Van den Broeck, Stefano Soatto. SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning. CoRL 2020
    - Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, Antonio M. López. Multimodal End-to-End Autonomous Driving. arXiv 1906.03199, 2019
questions: Based on the weaknesses mentioned above, the following experiments are required to verify the claims in the paper:
- Trajectory-only model with feature loss applied to each prediction timestep and auxiliary loss (on speed and value prediction) used as well. This requires training a new model and should be compared in Table 3.
- A single GRU module to predict both future waypoints and multi-timestep control predictions from the hidden state of the GRU. This requires training a new model and should be compared in Table 3.
- Direct ensemble (average control predictions) of control-only and trajectory-only models in Table 4. This does not require training any model.
- An ensemble of multi-step control prediction and trajectory prediction (both situation-based fusion and direct averaging should be considered) without the trajectory-guided attention module. This does not require training any model and should be compared in Table 4.

Certain parts need clarification:
- L194-195: How does the multi-step control prediction branch mitigate the IID assumption?
- L202-204: How does the trajectory branch improve the ability to incorporate static layout information (eg. curbs)?
- L309-L313: Why does TCP has lower route completion compared to LiDAR-based methods. Is it because the TCP gets blocked more often than other methods? It is also possible that taking the maximum of brake values from different models for the ensemble (L33-34 in supplementary) leads to an overly conservative policy.
- The evaluation protocol for the experiments in Table 2,3,4 is different than Table 1 which uses the CARLA leaderboard. The authors should include a brief description of this evaluation protocol in the paper.
 - Fig. 1(a) shows a failure case at turnings for trajectory-based methods. Since the details of the PID controllers are not provided in the paper, it is unclear if the PID controller could just be tuned to better follow the trajectory.
 - Implementation details regarding the GRU modules, MLPs in the trajectory-guided attention, and weights for loss terms (Eq. 4,5,6) should also be included in the paper.

Additional suggestions to improve the paper:
- As mentioned earlier, it would be helpful if the authors could also provide comparisons to FASNet. This would greatly improve the paper.
- It would be interesting to visualize the GradCAM (Selvaraju et al. IJCV 2019) attention maps for the multi-step control prediction branch without the trajectory-guided attention module. This could provide useful insights into whether the multi-step control prediction helps the model to focus on important regions for future control predictions. These GradCAM attention maps can be compared to attention maps in Fig. 4 to verify the utility of trajectory-guided attention.
limitations: The authors have provided a discussion on limitations and societal impact in the supplementary. Additional suggestions are mentioned above.  

### Update after Rebuttal ###
I appreciate additional ablations and discussions with authors which helped me get a better understanding of the paper. After rebuttal and discussion with other reviewers, I am retaining my original score. 

My main concern is that majority of the gain is coming from the heuristic $\alpha$ used for ensembling. Table 2 in the main paper shows Control-only=`32.45`, trajectory-only=`28.29`. In one of the ablations provided by the authors in the rebuttal, ensemble of control-only & trajectory-only methods with non-constant $\alpha$ gives a score of `50.98`. TCP has a score of `57.01` in Table 4 in the main paper. The ensemble heuristic leads to a gain of around `19` points whereas all the architectural modifications - shared backbone, multi-step control, temporal module, trajectory-guided attention - combined result in a gain of `6` points. 

While it might be possible to use the fusion mechanism in a general and flexible way (eg. as a mixture of experts or using probabilistic uncertainty, as mentioned by the authors in the rebuttal), the paper currently uses a heuristic for fusion. Based on this, I agree with reviewer `6yad` that the paper does not have sufficient technical contribution for NeurIPS, and venues like IROS, ICRA are a better fit. The authors have mentioned multiple interesting directions, eg. link between the temporal module and world models, using mixture of experts or probabilistic uncertainty for fusion. I encourage the authors to explore these directions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents TCP, a novel trajectory predicting/controlling network architecture for vision-based end-to-end autonomous driving. At test time, TCP ensembles the control outputs from a direct prediction branch and a trajectory prediction  w/ PID controllers. Both branches use GRU to predict waypoints/controls auto-regressively and are jointly trained. At each timestep, the control prediction branch additionally takes as input attention-weighted image backbone features computed from the trajectory prediction branch.

strengths_and_weaknesses: === Strengths ===
+ The strongest aspect of this paper is its impressive performance on the challenging CARLA leaderboard. The fact that TCP is capable of attaining such a strong performance with a simple vision backbone is inspiring, suggesting prior methods for end-to-end driving in CARLA might all have suboptimal controller designs. 
+ The ablation studies are mostly comprehensive, and the design choices are well justified. 
+ The attention map visualization is insightful and shows what areas are attended for the control prediction task.

=== Weaknesses ===
- I do not have any major complaints regarding the technical content. My main question is how much performance gain comes from fusing the trajectory/control prediction branches as the title and paper implies, or suboptimal PID parameters for the trajectory-only branch, and the trajectory-attended image features inputs in the autoregressive prediction stage. I would like to see a trajectory-only variant where, similar to the proposed situation-based fusion, has situation-based PID controller parameters (different PID gains based on different high-level commands, or whether it’s turning etc.), which still has the novel trajectory-computed attention fusion module (i.e. the attention-weighed image features get consumed by the trajectory branch instead of the control branch). This will further help distinguish the main source of improvements among the design choices.

questions: Apart from the ablation mentioned in the Strengths & Weaknesses section, I would like to hear from the authors how they think TCP will work in a real world setting, what might be the main obstacles and bottlenecks that the proposed method could have when scaling up to real world data.

limitations: Same as my question in the section above – I would like to hear from the authors what they think might be the limitations of the proposed method when scaling to the real world.
review_title: null

Meta-review: metareview: The paper got split reviews: 1x reject, 1x borderline reject, 1x weak accept, 1x accept. All reviewers found the impressive performance on the challenging CARLA leaderboard to be a major strength of the paper.

Reviewer concerns stem from two factors: a) not enough technical contribution to warrant publication at NeurIPS (but results are still publication worthy at more domain-specific conferences eg ICRA, IROS), and b) bulk of the impressive performance (19 points) coming from the ensembling heuristic and only 6 points coming from proposed architectural modifications (shared backbone, multi-step control, temporal module and trajectory guided attention).

The meta-reviewer read through the paper, the reviews, the author response, and reviewer discussion. For the meta-reviewer, the impressiveness of the empirical results on a well-studied and important benchmark dominates the above reviewer concerns. As long as there is clear attribution and some understanding as to where this impressive performance improvement is coming from, the community will benefit from being aware of the results even though the proposed method may not be as technically deep as typical NeurIPS papers. The authors are encouraged to include the additional experiments conducted during the rebuttal phase into the final version of the paper, in particular the ones that help distill out the contribution of the different parts of the proposed system.
confidence: Certain
Decision: Accept