Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a fully differentiable yet modular pipeline for autonomous vehicle perception and control. It preserves the traditional abstractions such as perception, prediction, planning, and control but implements differentiable versions of these components so that gradients can pass all the way from task performance back to perception. They demonstrate the value of this on a trajectory prediction task. 
strengths_and_weaknesses: Strengths:

 - The idea is well-motivated and the approach seems well executed. The approach preserves some of the important interpretability charateristics of traditional methods while still enabling end-to-end learning. 
 - Using task performance to learn perceptual models is a promising area of research, and this paper showcases a possible method for doing so in the case of autonomous vehicles. 


Weaknesses / Questions:

 - The authors should highlight to what degree the differentiable components are novel. To my understanding they have mostly taken existing components and assemble them together so the contribution should be mostly considered at the system level. If this is not the case the authors should clarify. 

 - The cost function in (3) contains a lot of weights. How sensitive is the algorithm to the selection of these weights? Similar question for $\alpha_{1:3}$ in the combined loss function.

 - How is the discretization of $s$ chosen for the planner (i.e., what is the dimension of the Categorical distribution and why)?

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the work presents a novel autonomous vehicle pipeline that is modular yet differentiable. Although none of the individual components seems to be particularly novel, the system that is constructed is and the advantages that it affords seem to be convincing.  
issues: See above.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Rather than building an AV stack in a modular fashion, this work introduces a differentiable and modular stack for prediction, planning and control. This enables optimization of upstream components such as prediction via backprop through planning and control.
strengths_and_weaknesses: Strengths:
- The method is still interpretable in the sense that it is composed of modules with specific purposes. 
- The method is able to be trained end-to-end due to being fully differentiable. 
- Comparison to a non-differentiable but modular method is made clearly.

Weakness:
- no real world or even simulator results.
- not differentiable to all possible parameters
- Comparison to an end-to-end neural network is not made. The authors note that not all parameters are differentiable. How does this system compare to an end-to-end neural network that doesn't attempt to provide an interpretability? How much performance is being sacrificed for interpretability? Can the authors provide quantitative or qualitative examples of how this interpretability is useful?




robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Paper is well written and well motivated. 
The experimental section could be made much strong through additional baselines (particulary a non-modularized end-to-end system), and real world or even simulated experiments.
issues: - The argument for a fully data-driven yet modular and interpretable AV architecture is compelling in general, but could be better articulated in this work. Particularly there is no comparison or demonstration of their approach providing a benefit  over a non-interpretable end-to-end system.  It would be great if the authors could find some way to quantify this, or at least qualitatively demonstrate the difference.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an end-to-end differentiable stack for autonomous driving. The prediction module is a neural network, and the planning and control modules are hand-designed algorithms. Notably, the hand-designed algorithms are differentiable, which allows training of the upstream prediction module for a downstream control objective by backpropagating gradients through these hand-designed algorithms. The authors make planning differentiable by replacing the argmin operation with sampling from a categorical distribution, and the authors make use of an off-the-shelf differentiable MPC algorithm to make control differentiable. The authors show open-loop, offline results which indicate that this kind of planning-aware training of the prediction module improves performance. 

strengths_and_weaknesses: Strengths
- A step towards end-to-end training of prediction modules, while retaining the interpretability / stability of hand-designed control and planning algorithms. 
- The paper is well-written and easy to parse. 

Weaknesses
- Only offline / open-loop evaluation, so it is difficult to predict what real world performance will be like. 
- I found the metrics to be somewhat hard to interpret, but this could be because I’m not a researcher in the AV space.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I do not conduct research in autonomous driving, so take my recommendation with a grain of salt (this is the main reason I selected weak accept instead of strong accept). Overall, I felt the paper was well-motivated and executed, and tackles what I believe is an important problem. I recommend acceptance. 
issues: N/A
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors tackle the task of offering a full autonomous driving software stack. In contrast to classical approaches that focus on modular software containing of perception, planning and control, the authors present here software components that are full differentiable. The authors focus on prediction, planning and control modules that are made differentiable, train it in a simulation environment (nuScenes) and then show the results in comparison to baselines. The authors display that this approach - in contrast to prior End-to-End methods - gives more modularity and interpretability.
strengths_and_weaknesses: Strengths:

-The paper is well-written and good to read, the technical explanations are exceptional
- The paper tackles a very hot topic in the field, especially for AVs were we are currently seeing bottlenecks in hand-engineered algorithms
- Honest opinion about the limitations of this approach, it is not a full full differentiable stack
- The results indicate that "there is value in moving from purely prediction-oriented evaluation metrics towards downstream task-oriented metrics" -> This was new to me and is a significant result, especially with all the work in prediction-algorithms right now

Weaknesses:
- No hardware experiments were provided
- No Training results were provided. Although it is stated that the nuScens datasets is used there is nothing explained about training quality and training time
- No comparison to a modular software stack (classical pipeline) or a complete end-to-end pipeline. I do not understand the comparison with the "baseline" (We compare DiffStack to the same prediction-planning-control stack, but with modules
219 trained for different objectives that capture a varying degree of planning-awareness) ??
- The authors claim with the title its a full-stack but in the limitations, they show that this approach is only performed in open-loop training and evaluation
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: For me this paper is a strong accept since the paper contains many components and ideas that will potentially have a major impact in the field of autonomous vehicles. The paper is both well-written and technically deep, the authors explain whats going on in their different Software Stack Modules (Predction, Planning, Control) and how they are training in and end-to-end fashion. Generally that paper contains all important components and is of high interest for the community
issues: The biggest issue i have with the paper are the results and explanations of the outcome and why diffstack is better. Since the baseline is also diffstack but trained with different objectives (correct?) I do not see the justification for the discussion. The discussion section is great and shows why Diffstack is good, but I miss the comparison to a classical stack or a real end-to-end pipeline
review_title: null

Meta-review: metareview: This paper proposes a modular but fully differentiable stack for self-driving cars. The differentiability enables the gradient to back-propagate from task performance all the way to the prediction module. It demonstrates better prediction accuracy than the traditional modular approach while preserving interpretability and reusability.

All the reviewers agree that the paper is well motivated, written and executed. The paper tackles an important problem and contains ideas that will potentially have a major impact in the field of autonomous vehicles. Also, there are also a few areas for improvements brought up by the reviewers:
1) Validation with a close-loop simulation or real-world experiments would significantly improve the quality and the potential impact of this paper.
2) Comparison with a classical modular approach and/or a non-interpretable end-to-end system may reveal more benefits or limitations of this work. Such comparisons are worth adding and discussing.

The authors' response and the additional experiments have sufficiently addressed the main concerns in the original reviews. Thus, we would like to recommend accepting this paper.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)